Oct 30 03:02:16.780070 ip-10-0-134-92 systemd[1]: Starting Kubernetes Kubelet... Oct 30 03:02:17.218398 ip-10-0-134-92 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:17.218398 ip-10-0-134-92 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 03:02:17.218398 ip-10-0-134-92 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:17.218398 ip-10-0-134-92 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:17.225538 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225454 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.225538 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225535 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.225538 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225539 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.225538 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225542 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225548 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225552 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225555 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225558 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225561 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225563 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225566 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225569 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225571 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225574 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225577 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225579 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225582 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225585 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225587 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225590 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225601 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225604 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.225694 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225607 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225609 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225612 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225615 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225618 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225621 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225623 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225626 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225628 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225631 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225633 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225636 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225638 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225641 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225643 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225648 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225651 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225653 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225656 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225659 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.226155 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225661 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225664 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225666 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225669 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225671 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225674 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225677 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225679 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225681 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225684 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225687 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225689 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225692 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225695 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225697 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225700 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225702 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225705 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225708 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225711 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.226621 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225713 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225716 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225719 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225721 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225725 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225727 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225730 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225733 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225736 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225739 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225741 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225744 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225747 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225765 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225769 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225773 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225775 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225778 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225781 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225784 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.227115 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225786 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225790 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225793 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225796 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225798 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225801 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225803 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225806 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225808 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225811 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225814 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225816 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225819 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225822 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225824 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225827 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225830 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225832 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225835 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.227579 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225838 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225840 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225843 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225848 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.225851 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226625 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226632 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226635 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226637 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226640 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226643 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226645 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226648 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226651 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226653 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226657 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226660 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226663 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226665 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.228070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226668 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226670 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226683 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226686 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226689 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226692 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226694 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226697 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226700 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226703 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226705 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226708 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226710 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226713 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226716 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226719 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226721 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226723 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226726 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.228520 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226728 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226731 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226734 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226736 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226738 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226741 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226743 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226745 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226766 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226770 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226773 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226776 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226779 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226781 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226783 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226786 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226789 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226792 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226794 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226797 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.228986 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226800 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226803 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226805 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226808 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226811 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226813 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226816 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226819 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226821 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226824 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226828 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226832 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226835 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226838 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226841 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226844 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226846 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226850 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226852 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.229447 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226855 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226857 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226860 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226862 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226865 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226868 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226871 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226873 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226878 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226881 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226885 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226889 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226892 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226895 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226898 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226900 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226903 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226905 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226908 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.229904 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226911 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226913 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226916 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226918 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226921 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226923 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226926 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226928 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226931 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226934 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226936 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226939 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226941 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226944 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.226946 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227438 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227443 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227447 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227450 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.230348 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227453 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227455 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227458 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227461 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227464 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227466 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227469 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227472 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227474 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227477 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227479 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227481 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227484 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227486 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227489 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227491 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227494 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227496 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227499 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.230828 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227501 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227504 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227506 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227509 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227511 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227513 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227516 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227518 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227521 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227523 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227525 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227528 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227531 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227533 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227536 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227538 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227541 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227543 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227545 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227548 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.231275 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227551 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227553 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227556 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227558 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227561 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227563 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227566 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227569 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227571 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227574 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227576 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227579 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227581 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227584 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227586 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227589 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227592 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227594 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227597 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.231771 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227599 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227603 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227607 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227611 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227614 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227617 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227621 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227623 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227626 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227628 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227631 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227634 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227636 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227639 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227642 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227644 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227646 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227649 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227651 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.232214 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227654 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227656 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227659 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227661 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227664 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227666 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227669 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227672 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227676 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227679 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227682 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227685 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227687 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227690 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227692 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227695 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227697 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227700 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227702 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.232692 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227704 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227707 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227710 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227713 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227716 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.227721 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228516 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228525 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228531 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228537 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228541 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228545 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228549 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228553 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228556 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228559 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228563 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228567 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228571 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228573 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228576 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228579 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228582 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 03:02:17.233138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228585 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228590 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228593 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228596 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228599 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228602 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228606 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228609 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228612 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228615 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228618 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228624 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228627 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228630 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228635 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228639 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228642 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228645 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228647 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228650 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228653 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228658 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228661 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228664 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 03:02:17.233766 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228667 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228669 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228674 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228676 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228680 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228683 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228686 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228689 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228692 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228694 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228697 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228700 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228703 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228707 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228710 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228713 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228716 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228719 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228722 2574 flags.go:64] FLAG: --help="false" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228727 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228730 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228733 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228737 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228741 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 03:02:17.234342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228744 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228747 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228768 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228771 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228775 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228778 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228781 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228784 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228787 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228790 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228793 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228796 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228798 2574 flags.go:64] FLAG: --lock-file="" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228801 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228804 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228807 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228812 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228815 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228819 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228821 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228824 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228827 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228830 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228833 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228837 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 03:02:17.234926 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228840 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228845 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228849 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228853 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228856 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228861 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228864 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228866 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228869 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228876 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228879 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228882 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228885 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228888 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228891 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228894 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228897 2574 flags.go:64] FLAG: --port="10250" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228900 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228903 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02f043e8c8fec38fc" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228906 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228909 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228911 2574 flags.go:64] FLAG: --register-node="true" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228915 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228918 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228921 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228924 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 03:02:17.235533 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228926 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228930 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228933 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228936 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228939 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228941 2574 flags.go:64] FLAG: --runonce="false" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228944 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228947 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228951 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228954 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228957 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228962 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228965 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228969 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228971 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228974 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228977 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228980 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228983 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228986 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228989 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228994 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.228997 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229000 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229004 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 03:02:17.236195 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229007 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229010 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229013 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229016 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229019 2574 flags.go:64] FLAG: --v="2" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229023 2574 flags.go:64] FLAG: --version="false" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229027 2574 flags.go:64] FLAG: --vmodule="" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229031 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229034 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229165 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229169 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229172 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229175 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229177 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229182 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229188 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229191 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229194 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229198 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229201 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229204 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.236820 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229207 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229209 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229212 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229214 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229217 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229220 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229222 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229225 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229227 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229229 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229232 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229234 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229237 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229239 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229242 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229244 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229247 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229249 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229251 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229254 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.237320 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229256 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229259 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229261 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229264 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229266 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229269 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229274 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229276 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229279 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229281 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229284 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229287 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229290 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229292 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229294 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229297 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229299 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229302 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229304 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229306 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.237802 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229309 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229311 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229314 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229316 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229319 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229321 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229324 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229326 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229328 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229331 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229333 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229335 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229338 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229340 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229342 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229345 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229348 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229350 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229354 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.238278 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229356 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229359 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229361 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229364 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229366 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229371 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229374 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229377 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229380 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229383 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229385 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229387 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229390 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229392 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229395 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229397 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229399 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229402 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229404 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229407 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.238735 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229409 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229412 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229414 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229417 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229419 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229421 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229424 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229426 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229428 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229431 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229433 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229437 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229440 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229442 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229445 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.229471 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:17.239212 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229506 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229511 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229514 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229517 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229520 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229523 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229526 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229528 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229531 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229533 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229536 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229539 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229541 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229544 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229546 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229548 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229551 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229553 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229556 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229558 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.239583 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229560 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229563 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229565 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229568 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229570 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229574 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229576 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229580 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229582 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229585 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229587 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229590 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229592 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229595 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229597 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229601 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229603 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229606 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229608 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229611 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.240054 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229613 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229616 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229618 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229620 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229623 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229626 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229628 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229630 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229633 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229635 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229638 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229640 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229642 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229645 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229647 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229650 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229652 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229656 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229658 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.240514 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229662 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229664 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229666 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229669 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229672 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229674 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229678 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229681 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229684 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229687 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229690 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229693 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229695 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229697 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229700 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229702 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229705 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229707 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229710 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229713 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.240979 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229716 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229718 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229721 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229723 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229725 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229728 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229730 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229732 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229735 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229737 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229741 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229743 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229747 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229765 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229770 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229772 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229775 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229777 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229780 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229782 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.241470 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229785 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229789 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229794 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229797 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229800 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229802 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.229805 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.237062 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.237075 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.237810 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.237816 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237889 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237895 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237898 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237902 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237905 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237908 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237910 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.241952 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237913 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237916 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237918 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237921 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237924 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237926 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237929 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237931 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237934 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237937 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237939 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237942 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237944 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237947 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237949 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237951 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237955 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237957 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237960 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237962 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.242382 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237965 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237968 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237972 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237974 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237977 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237980 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237982 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237985 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237987 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237990 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237993 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237996 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.237999 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238001 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238004 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238006 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238009 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238012 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238014 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.242879 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238017 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238019 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238022 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238024 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238027 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238029 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238032 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238034 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238037 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238039 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238042 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238045 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238047 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238051 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238056 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238059 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238062 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238065 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238067 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.243301 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238070 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238072 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238075 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238077 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238080 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238082 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238085 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238087 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238090 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238092 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238095 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238097 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238099 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238102 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238104 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238107 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238109 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238111 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238114 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238116 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.243723 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238119 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238121 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238124 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238127 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238130 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238132 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238136 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238140 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238143 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238146 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238149 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238151 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238154 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238157 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238159 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238162 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238164 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238167 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238169 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.244200 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238172 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238174 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.238203 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238399 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238404 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238407 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238409 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238412 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238415 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238417 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238420 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238423 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238425 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238428 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238430 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238433 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.244631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238435 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238438 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238440 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238443 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238446 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238448 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238451 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238453 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238455 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238458 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238461 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238463 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238465 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238468 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238470 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238473 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238475 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238478 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238481 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.245070 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238485 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238488 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238492 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238495 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238498 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238501 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238504 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238507 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238509 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238512 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238514 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238517 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238519 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238522 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238524 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238526 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238529 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238532 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238535 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238537 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.245540 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238540 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238542 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238545 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238547 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238550 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238552 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238554 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238557 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238559 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238561 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238564 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238566 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238569 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238571 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238574 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238576 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238579 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238581 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238583 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238588 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.246154 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238590 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238593 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238595 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238598 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238600 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238603 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238605 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238607 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238610 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238618 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238621 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238623 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238626 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238628 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238630 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238633 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238635 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238638 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238640 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238642 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.246608 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238645 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238647 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238650 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238653 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238656 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238659 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238661 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238664 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238666 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238669 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238680 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238685 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238688 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238690 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.238714 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238780 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:17.247242 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238785 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238789 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238792 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238795 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238798 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238801 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238809 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238812 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238815 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238817 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238819 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238822 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238825 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238827 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238830 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238832 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238834 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238837 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238839 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:17.247647 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238842 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238844 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238849 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238853 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238857 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238861 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238864 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238868 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238871 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238874 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238877 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238880 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238883 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238885 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238888 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238890 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238892 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238895 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:17.248218 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238897 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238906 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238909 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238911 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238913 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238916 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238918 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238921 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238923 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238925 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238928 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238930 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238932 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238935 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238937 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238940 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238942 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238944 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238947 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:17.248651 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238949 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238952 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238955 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238958 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238960 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238963 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238965 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238968 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238970 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238972 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238975 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238977 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238980 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238983 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238985 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238993 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238996 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.238998 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239000 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239003 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:17.249106 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239005 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239007 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239010 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239012 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239015 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239017 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239019 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239022 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239024 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239028 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239031 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239034 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239037 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239039 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239043 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239045 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239048 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239050 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239053 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:17.249598 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239055 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239057 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239060 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239062 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239064 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239067 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239070 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239077 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:17.239080 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.239133 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.241577 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.242633 2574 server.go:1014] "Starting client certificate rotation" Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.242727 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.242774 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.245665 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:17.250052 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.247872 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:17.260367 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.260352 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 03:02:17.264584 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.264569 2574 log.go:25] "Validated CRI v1 image API" Oct 30 03:02:17.265467 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.265436 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 03:02:17.269922 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.269894 2574 fs.go:135] Filesystem UUIDs: map[2d996a21-7135-4f1f-a419-a85ee8c6676e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e022853c-0f66-4f2b-89a4-458f12017129:/dev/nvme0n1p4] Oct 30 03:02:17.269998 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.269921 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 03:02:17.276559 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.276456 2574 manager.go:217] Machine: {Timestamp:2026-10-30 03:02:17.274765861 +0000 UTC m=+0.381612746 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3022946 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2039908a88285523389463daa3ecc4 SystemUUID:ec203990-8a88-2855-2338-9463daa3ecc4 BootID:0545f8f1-3a29-4e95-8c0d-62f99f24f1f7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:67:cb:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:67:cb:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:75:02:1a:19:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 03:02:17.276559 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.276557 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 03:02:17.276713 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.276630 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 03:02:17.279262 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.279242 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 03:02:17.279382 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.279264 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 03:02:17.279429 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.279385 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 03:02:17.279429 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.279394 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 03:02:17.279429 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.279419 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:17.279500 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.279436 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:17.280831 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.280821 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 03:02:17.284028 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.284015 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 03:02:17.284145 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.284135 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 03:02:17.284942 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.284932 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 03:02:17.284987 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.284976 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 03:02:17.285029 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.284989 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 03:02:17.285029 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.285002 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 03:02:17.285029 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.285015 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 03:02:17.285384 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.285369 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:17.288055 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.288020 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 03:02:17.295830 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.295817 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:17.296716 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296705 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 03:02:17.296775 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296735 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 03:02:17.296912 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296901 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 03:02:17.296940 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296917 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 03:02:17.296940 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296924 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 03:02:17.296940 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296930 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 03:02:17.296940 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296935 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 03:02:17.296940 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.296940 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 03:02:17.297646 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.297638 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 03:02:17.297673 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.297648 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 03:02:17.297673 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.297655 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 03:02:17.297673 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.297660 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 03:02:17.297673 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.297669 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 03:02:17.297809 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.297678 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 03:02:17.298462 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.298453 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 03:02:17.298493 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.298463 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 03:02:17.302734 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.302720 2574 server.go:1263] "Started kubelet" Oct 30 03:02:17.302821 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.302777 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 03:02:17.302901 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.302856 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 03:02:17.302953 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.302920 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 03:02:17.303582 ip-10-0-134-92 systemd[1]: Started Kubernetes Kubelet. Oct 30 03:02:17.309668 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.308068 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 03:02:17.309668 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.309370 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 03:02:17.313421 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.313400 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 03:02:17.316626 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.316601 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 03:02:17.317326 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.317308 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 03:02:17.317497 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.317472 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:17.318091 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318074 2574 factory.go:55] Registering systemd factory Oct 30 03:02:17.318174 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318094 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 03:02:17.318174 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318134 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 03:02:17.318280 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318263 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 03:02:17.318339 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318284 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 03:02:17.318339 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318315 2574 factory.go:153] Registering CRI-O factory Oct 30 03:02:17.318339 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318327 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 03:02:17.318483 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318379 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 03:02:17.318483 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318395 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 03:02:17.318483 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318403 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 03:02:17.318483 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318396 2574 factory.go:103] Registering Raw factory Oct 30 03:02:17.318483 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.318425 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 03:02:17.319454 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.319440 2574 manager.go:319] Starting recovery of all containers Oct 30 03:02:17.326511 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.326482 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 03:02:17.327627 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.326490 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-92.ec2.internal.18e32f87e333835a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-92.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-92.ec2.internal,},FirstTimestamp:2026-10-30 03:02:17.302483802 +0000 UTC m=+0.409330688,LastTimestamp:2026-10-30 03:02:17.302483802 +0000 UTC m=+0.409330688,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-92.ec2.internal,}" Oct 30 03:02:17.329418 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.329402 2574 manager.go:324] Recovery completed Oct 30 03:02:17.333331 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.333316 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:17.335030 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.335011 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:17.335115 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.335042 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:17.335115 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.335054 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:17.335544 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.335533 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 03:02:17.335544 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.335542 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 03:02:17.335627 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.335545 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:17.335627 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.335558 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 03:02:17.336226 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.336211 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:17.336294 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.336240 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:17.336294 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.336257 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:17.336595 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.336582 2574 policy_none.go:50] "Start" Oct 30 03:02:17.336632 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.336598 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 03:02:17.336632 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.336608 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 03:02:17.337501 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.337490 2574 policy_none.go:44] "Start" Oct 30 03:02:17.340572 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.340552 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 03:02:17.340634 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.340601 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gq5p9" Oct 30 03:02:17.340665 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.340652 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gq5p9" Oct 30 03:02:17.343957 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.343932 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 03:02:17.344648 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.344626 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 03:02:17.344648 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.344627 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 03:02:17.371811 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.371673 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:17.377401 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.377375 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 03:02:17.379679 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.379665 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 03:02:17.379743 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.379692 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 03:02:17.379743 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.379700 2574 server.go:87] "Starting device plugin registration server" Oct 30 03:02:17.379936 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.379924 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 03:02:17.379991 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.379936 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 03:02:17.380039 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.380023 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 03:02:17.380107 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.380089 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 03:02:17.380107 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.380103 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 03:02:17.380626 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.380609 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 03:02:17.380692 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.380654 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:17.455546 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.455512 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 03:02:17.457607 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.456823 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 03:02:17.457607 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.456842 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 03:02:17.457607 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.456862 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 03:02:17.457607 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.456904 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 03:02:17.471354 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.471314 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:17.480561 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.480547 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:17.481377 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.481364 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:17.481447 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.481392 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:17.481447 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.481406 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:17.481447 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.481430 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.500077 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.500061 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.500128 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.500084 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-92.ec2.internal\": node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:17.537727 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.537709 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:17.558105 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.558086 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal"] Oct 30 03:02:17.558157 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.558141 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:17.558913 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.558896 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:17.558976 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.558925 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:17.558976 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.558935 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:17.561111 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561099 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:17.561288 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.561288 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561290 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:17.561777 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561743 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:17.561856 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561776 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:17.561856 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561789 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:17.561856 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561795 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:17.561856 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561801 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:17.561856 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.561806 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:17.563936 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.563921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.564011 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.563953 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:17.564618 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.564604 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:17.564694 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.564632 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:17.564694 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.564646 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:17.590203 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.590187 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-92.ec2.internal\" not found" node="ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.592657 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.592640 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-92.ec2.internal\" not found" node="ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.619805 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.619784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.619880 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.619809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.619880 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.619828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c18c0405adfb2995cb6b447bf5eb736-config\") pod \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" (UID: \"7c18c0405adfb2995cb6b447bf5eb736\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.637900 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.637878 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:17.720477 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.720453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.720572 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.720479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.720572 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.720498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c18c0405adfb2995cb6b447bf5eb736-config\") pod \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" (UID: \"7c18c0405adfb2995cb6b447bf5eb736\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.720572 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.720549 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c18c0405adfb2995cb6b447bf5eb736-config\") pod \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" (UID: \"7c18c0405adfb2995cb6b447bf5eb736\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.720682 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.720564 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.720682 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.720564 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.738409 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.738360 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:17.838561 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.838536 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:17.892764 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.892731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.895607 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:17.895587 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:17.939197 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:17.939174 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:18.039683 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.039624 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:18.140176 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.140150 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:18.178031 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.178009 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:18.218084 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.218062 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:18.240523 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.240501 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:18.242259 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.242244 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:18.242361 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.242348 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 03:02:18.242445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.242427 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Oct 30 03:02:18.242495 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.242438 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Oct 30 03:02:18.242495 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.242429 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Oct 30 03:02:18.242495 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.242438 2574 kubelet.go:3370] "Failed creating a mirror pod" err="Post \"https://a78c045a7c98c483f9b286696453a865-1125b8a5de198092.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/kube-system/pods\": write tcp 10.0.134.92:59610->3.136.126.54:6443: use of closed network connection" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:18.285660 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.285638 2574 apiserver.go:52] "Watching apiserver" Oct 30 03:02:18.317167 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.317141 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 03:02:18.345531 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.345486 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:57:17 +0000 UTC" deadline="2028-08-02 14:20:00.435643146 +0000 UTC" Oct 30 03:02:18.345531 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.345517 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15419h17m42.090129117s" Oct 30 03:02:18.364399 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.364374 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 03:02:18.364669 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.364651 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-j2q75","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal","openshift-multus/multus-additional-cni-plugins-9fhjj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7","openshift-cluster-node-tuning-operator/tuned-gnxzn","openshift-network-diagnostics/network-check-target-6szch","kube-system/global-pull-secret-syncer-m266j","openshift-ovn-kubernetes/ovnkube-node-qtbgx","openshift-dns/node-resolver-jj8ks","openshift-image-registry/node-ca-shbmk","openshift-multus/multus-tpl8m","openshift-multus/network-metrics-daemon-qxx87","openshift-network-operator/iptables-alerter-hq9jd"] Oct 30 03:02:18.366942 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.366925 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.369579 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.369555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.373507 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.371915 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.374739 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.374722 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.376891 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.376873 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.379020 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.379002 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:18.379098 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.379070 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:18.381700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.381680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:18.381815 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.381741 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:18.383587 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.383572 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7q2t\"" Oct 30 03:02:18.384502 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.384483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.386337 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.386317 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 03:02:18.386416 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.386347 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 03:02:18.387431 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.387418 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.387484 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.387469 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:18.390415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.390397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.392826 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.392808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.394935 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.394922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.400734 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.400716 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:18.400925 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.400909 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402048 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402059 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402068 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402077 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402060 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402096 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402105 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402118 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402127 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 03:02:18.402141 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402134 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 03:02:18.403012 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.402899 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 03:02:18.403296 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.403281 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 03:02:18.403380 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.403311 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r9jr8" Oct 30 03:02:18.404346 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.404330 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 03:02:18.418982 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.418966 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 03:02:18.423672 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-etc-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.423822 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-node-log\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.423822 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wttlq\" (UniqueName: \"kubernetes.io/projected/dde3589d-5a4c-4176-b493-f5ecfa81349f-kube-api-access-wttlq\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.423822 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423768 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-bin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.423822 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-multus\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.423822 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-kubelet\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423849 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-socket-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423909 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.423982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-config\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckj96\" (UniqueName: \"kubernetes.io/projected/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-kube-api-access-ckj96\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424056 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.424088 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlh9\" (UniqueName: \"kubernetes.io/projected/9e9460d2-8d0f-4aca-a2a0-860137f1b801-kube-api-access-wwlh9\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424104 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf6gp\" (UniqueName: \"kubernetes.io/projected/20788c87-fbba-4a1c-9d38-f32396e8ed24-kube-api-access-cf6gp\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bldd\" (UniqueName: \"kubernetes.io/projected/cc68d015-081c-41bc-b7ec-2443b5693c76-kube-api-access-6bldd\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-cnibin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-socket-dir-parent\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-ovn\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424303 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-kubernetes\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-var-lib-kubelet\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.424415 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424381 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-host\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-kubelet-config\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424461 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f7ec66-f78a-405d-95c0-47a26e24a69d-host\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-k8s-cni-cncf-io\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-hostroot\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424580 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-multus-certs\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424681 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-netd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-cni-binary-copy\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-systemd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.424853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-etc-kubernetes\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424881 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cnibin\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-sys\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gssmr\" (UniqueName: \"kubernetes.io/projected/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-kube-api-access-gssmr\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.424991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-dbus\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovn-node-metrics-cert\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-script-lib\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425069 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-daemon-config\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.425119 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-agent-certs\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-konnectivity-ca\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20788c87-fbba-4a1c-9d38-f32396e8ed24-iptables-alerter-script\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20788c87-fbba-4a1c-9d38-f32396e8ed24-host-slash\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hq8\" (UniqueName: \"kubernetes.io/projected/454e1d2d-f058-4a2b-a420-298761f2ff6a-kube-api-access-57hq8\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425268 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-hosts-file\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425289 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50f7ec66-f78a-405d-95c0-47a26e24a69d-serviceca\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425324 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysconfig\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.425379 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-run\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.425619 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-tuned\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.425619 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425405 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-kubelet\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425619 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425427 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-sys-fs\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.425619 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-env-overrides\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425619 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszxf\" (UniqueName: \"kubernetes.io/projected/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-kube-api-access-fszxf\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425619 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmpmr\" (UniqueName: \"kubernetes.io/projected/50f7ec66-f78a-405d-95c0-47a26e24a69d-kube-api-access-fmpmr\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.425619 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-conf\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.425862 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-device-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.425862 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-var-lib-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425862 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-netns\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.425862 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-system-cni-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.425862 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-lib-modules\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.425862 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-slash\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.425862 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425804 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-log-socket\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425896 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-bin\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-tmp\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-systemd-units\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.425979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-netns\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-system-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.426075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426053 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-modprobe-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.426332 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-systemd\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.426332 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-registration-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.426332 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-conf-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.426428 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-tmp-dir\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.426428 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:18.426584 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.426437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.436108 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.436074 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c18c0405adfb2995cb6b447bf5eb736.slice/crio-aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d WatchSource:0}: Error finding container aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d: Status 404 returned error can't find the container with id aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d Oct 30 03:02:18.436528 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.436509 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca45c9b1760cdbab09f6b1b711a4a17.slice/crio-cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb WatchSource:0}: Error finding container cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb: Status 404 returned error can't find the container with id cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb Oct 30 03:02:18.440274 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.440258 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:02:18.460421 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.460346 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" event={"ID":"7c18c0405adfb2995cb6b447bf5eb736","Type":"ContainerStarted","Data":"aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d"} Oct 30 03:02:18.461498 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.461475 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" event={"ID":"dca45c9b1760cdbab09f6b1b711a4a17","Type":"ContainerStarted","Data":"cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb"} Oct 30 03:02:18.484945 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.484925 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 03:02:18.485278 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485260 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j94bv\"" Oct 30 03:02:18.485356 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485309 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hhqhj\"" Oct 30 03:02:18.485508 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485491 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tc5sd\"" Oct 30 03:02:18.485598 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485583 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f9hw2\"" Oct 30 03:02:18.485643 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485589 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nc7qz\"" Oct 30 03:02:18.485643 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485603 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 03:02:18.485643 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485601 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 03:02:18.485643 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485621 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bl9p4\"" Oct 30 03:02:18.485643 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485602 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mpbq7\"" Oct 30 03:02:18.485816 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.485663 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6fpwt\"" Oct 30 03:02:18.488075 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488060 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 03:02:18.488380 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488367 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 03:02:18.488429 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488379 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 03:02:18.488429 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488386 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 03:02:18.488429 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488408 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 03:02:18.488527 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488441 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 03:02:18.488579 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488568 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 03:02:18.488629 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.488618 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 03:02:18.527298 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-cni-binary-copy\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.527354 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-systemd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.527354 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.527354 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-etc-kubernetes\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cnibin\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-sys\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527405 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-systemd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527425 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gssmr\" (UniqueName: \"kubernetes.io/projected/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-kube-api-access-gssmr\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527426 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-dbus\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527449 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cnibin\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovn-node-metrics-cert\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527476 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-sys\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.527492 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527428 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-etc-kubernetes\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-script-lib\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-daemon-config\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-agent-certs\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-konnectivity-ca\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20788c87-fbba-4a1c-9d38-f32396e8ed24-iptables-alerter-script\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20788c87-fbba-4a1c-9d38-f32396e8ed24-host-slash\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527640 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527664 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hq8\" (UniqueName: \"kubernetes.io/projected/454e1d2d-f058-4a2b-a420-298761f2ff6a-kube-api-access-57hq8\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527666 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-dbus\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527688 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-hosts-file\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50f7ec66-f78a-405d-95c0-47a26e24a69d-serviceca\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysconfig\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-run\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-tuned\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-kubelet\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527845 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-sys-fs\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.527975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527867 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-env-overrides\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszxf\" (UniqueName: \"kubernetes.io/projected/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-kube-api-access-fszxf\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527919 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpmr\" (UniqueName: \"kubernetes.io/projected/50f7ec66-f78a-405d-95c0-47a26e24a69d-kube-api-access-fmpmr\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-conf\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-device-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.527998 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-var-lib-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528048 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-var-lib-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528053 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528043 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-cni-binary-copy\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-netns\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528128 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysconfig\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-system-cni-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528152 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-sys-fs\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-lib-modules\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528171 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-system-cni-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528181 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-script-lib\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-slash\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.528700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20788c87-fbba-4a1c-9d38-f32396e8ed24-host-slash\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528219 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-log-socket\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-bin\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-tmp\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528323 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-conf\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-systemd-units\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-netns\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528386 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-slash\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-system-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528246 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-netns\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528434 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-system-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528434 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-daemon-config\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528075 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528482 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50f7ec66-f78a-405d-95c0-47a26e24a69d-serviceca\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528495 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-systemd-units\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528533 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-hosts-file\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.529445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528551 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-device-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528570 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528586 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-log-socket\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528587 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-netns\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528615 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528616 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-konnectivity-ca\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-modprobe-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528637 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-env-overrides\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528615 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-run\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-kubelet\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528681 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-bin\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-systemd\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528785 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-lib-modules\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528789 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-registration-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528815 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-modprobe-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528823 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-systemd\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-conf-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.530253 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528886 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-conf-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-tmp-dir\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528923 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-registration-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.528977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-etc-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-node-log\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529097 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20788c87-fbba-4a1c-9d38-f32396e8ed24-iptables-alerter-script\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttlq\" (UniqueName: \"kubernetes.io/projected/dde3589d-5a4c-4176-b493-f5ecfa81349f-kube-api-access-wttlq\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-bin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529266 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-tmp-dir\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-multus\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-kubelet\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529336 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529342 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-kubelet\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529357 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-bin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529381 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-node-log\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.531281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529385 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-multus\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529379 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-etc-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-socket-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529449 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-config\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529492 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-socket-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckj96\" (UniqueName: \"kubernetes.io/projected/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-kube-api-access-ckj96\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529531 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.529610 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlh9\" (UniqueName: \"kubernetes.io/projected/9e9460d2-8d0f-4aca-a2a0-860137f1b801-kube-api-access-wwlh9\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.529733 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:19.029714417 +0000 UTC m=+2.136561310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529769 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6gp\" (UniqueName: \"kubernetes.io/projected/20788c87-fbba-4a1c-9d38-f32396e8ed24-kube-api-access-cf6gp\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.529785 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:18.532106 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529796 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bldd\" (UniqueName: \"kubernetes.io/projected/cc68d015-081c-41bc-b7ec-2443b5693c76-kube-api-access-6bldd\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.529820 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret podName:90b350e9-6998-45b6-882a-17019e89bdf6 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:19.029809106 +0000 UTC m=+2.136655981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret") pod "global-pull-secret-syncer-m266j" (UID: "90b350e9-6998-45b6-882a-17019e89bdf6") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529837 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-cnibin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-socket-dir-parent\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-ovn\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-kubernetes\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.529960 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-var-lib-kubelet\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530049 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-cnibin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-host\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530162 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-kubelet-config\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530183 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-socket-dir-parent\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530229 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-var-lib-kubelet\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530233 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-host\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530267 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f7ec66-f78a-405d-95c0-47a26e24a69d-host\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530275 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-kubelet-config\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:18.532839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530325 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-ovn\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530334 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-kubernetes\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f7ec66-f78a-405d-95c0-47a26e24a69d-host\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530373 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-k8s-cni-cncf-io\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530402 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-k8s-cni-cncf-io\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530410 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-hostroot\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530436 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-multus-certs\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-netd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-config\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530566 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-netd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530583 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-hostroot\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530608 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-multus-certs\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.530785 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.531027 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.532402 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-tuned\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.532427 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovn-node-metrics-cert\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.533284 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.532451 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-agent-certs\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.533740 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.532831 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-tmp\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.555309 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.555288 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gssmr\" (UniqueName: \"kubernetes.io/projected/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-kube-api-access-gssmr\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.555385 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.555344 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hq8\" (UniqueName: \"kubernetes.io/projected/454e1d2d-f058-4a2b-a420-298761f2ff6a-kube-api-access-57hq8\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.558598 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.558578 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:18.558708 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.558605 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:18.558708 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.558617 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gt8qv for pod openshift-network-diagnostics/network-check-target-6szch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:18.558708 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.558639 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6gp\" (UniqueName: \"kubernetes.io/projected/20788c87-fbba-4a1c-9d38-f32396e8ed24-kube-api-access-cf6gp\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.558708 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:18.558689 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv podName:4d22ace0-adef-488a-a025-83b8270d25d9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:19.058658509 +0000 UTC m=+2.165505398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gt8qv" (UniqueName: "kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv") pod "network-check-target-6szch" (UID: "4d22ace0-adef-488a-a025-83b8270d25d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:18.559113 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.558881 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpmr\" (UniqueName: \"kubernetes.io/projected/50f7ec66-f78a-405d-95c0-47a26e24a69d-kube-api-access-fmpmr\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.560358 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.560331 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckj96\" (UniqueName: \"kubernetes.io/projected/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-kube-api-access-ckj96\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.560600 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.560582 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszxf\" (UniqueName: \"kubernetes.io/projected/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-kube-api-access-fszxf\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.560658 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.560645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlh9\" (UniqueName: \"kubernetes.io/projected/9e9460d2-8d0f-4aca-a2a0-860137f1b801-kube-api-access-wwlh9\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.560890 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.560872 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bldd\" (UniqueName: \"kubernetes.io/projected/cc68d015-081c-41bc-b7ec-2443b5693c76-kube-api-access-6bldd\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:18.561536 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.561494 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttlq\" (UniqueName: \"kubernetes.io/projected/dde3589d-5a4c-4176-b493-f5ecfa81349f-kube-api-access-wttlq\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.689509 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.689477 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:18.699654 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.699628 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436fce07_fd6a_4b3e_8b1a_9d2f9de6e6b7.slice/crio-d52adaf7656f9ec4b858a919f8a57863f076a7780a3e974ade6f2d36cfff4f16 WatchSource:0}: Error finding container d52adaf7656f9ec4b858a919f8a57863f076a7780a3e974ade6f2d36cfff4f16: Status 404 returned error can't find the container with id d52adaf7656f9ec4b858a919f8a57863f076a7780a3e974ade6f2d36cfff4f16 Oct 30 03:02:18.701342 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.701289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:18.708898 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.708876 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20788c87_fbba_4a1c_9d38_f32396e8ed24.slice/crio-37803cef26cb335d9c1ba97d96e97cdc36bf69aade2685e53418abdbda7ae7b8 WatchSource:0}: Error finding container 37803cef26cb335d9c1ba97d96e97cdc36bf69aade2685e53418abdbda7ae7b8: Status 404 returned error can't find the container with id 37803cef26cb335d9c1ba97d96e97cdc36bf69aade2685e53418abdbda7ae7b8 Oct 30 03:02:18.730169 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.730143 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:18.738266 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.738244 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9460d2_8d0f_4aca_a2a0_860137f1b801.slice/crio-6a8bb95189927aa9434203255cc30cacd1a98a6dd47b2bb7d1e54e4831856447 WatchSource:0}: Error finding container 6a8bb95189927aa9434203255cc30cacd1a98a6dd47b2bb7d1e54e4831856447: Status 404 returned error can't find the container with id 6a8bb95189927aa9434203255cc30cacd1a98a6dd47b2bb7d1e54e4831856447 Oct 30 03:02:18.738859 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.738840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:18.744552 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.744535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:18.746244 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.746226 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454e1d2d_f058_4a2b_a420_298761f2ff6a.slice/crio-38532d928dadb938ff3c311632a6402097546df91bbcfac7b98ed6cf9dd12754 WatchSource:0}: Error finding container 38532d928dadb938ff3c311632a6402097546df91bbcfac7b98ed6cf9dd12754: Status 404 returned error can't find the container with id 38532d928dadb938ff3c311632a6402097546df91bbcfac7b98ed6cf9dd12754 Oct 30 03:02:18.753003 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.752985 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655ddd5c_e5dd_4217_b597_6dfb5e33fa73.slice/crio-bdcb7479e6eb8bd1d6e34340b40cd357bf8f31453dd93f5b5cde5904659aae1d WatchSource:0}: Error finding container bdcb7479e6eb8bd1d6e34340b40cd357bf8f31453dd93f5b5cde5904659aae1d: Status 404 returned error can't find the container with id bdcb7479e6eb8bd1d6e34340b40cd357bf8f31453dd93f5b5cde5904659aae1d Oct 30 03:02:18.761943 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.761926 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:18.764126 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.764083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tpl8m" Oct 30 03:02:18.767882 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.767863 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:18.770144 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.770123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:18.772136 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.772115 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f7ec66_f78a_405d_95c0_47a26e24a69d.slice/crio-aa0bfb13c8c64fefa115073d60c87b4451f7b8a261dd3a332e37297dd4d42098 WatchSource:0}: Error finding container aa0bfb13c8c64fefa115073d60c87b4451f7b8a261dd3a332e37297dd4d42098: Status 404 returned error can't find the container with id aa0bfb13c8c64fefa115073d60c87b4451f7b8a261dd3a332e37297dd4d42098 Oct 30 03:02:18.774876 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.774818 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddde3589d_5a4c_4176_b493_f5ecfa81349f.slice/crio-216d60ca2d895fec6796ef693433e1acd4f050bfdce4b7a6ec31fc5be06c1489 WatchSource:0}: Error finding container 216d60ca2d895fec6796ef693433e1acd4f050bfdce4b7a6ec31fc5be06c1489: Status 404 returned error can't find the container with id 216d60ca2d895fec6796ef693433e1acd4f050bfdce4b7a6ec31fc5be06c1489 Oct 30 03:02:18.780631 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:18.780597 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585acb34_e959_4d7b_9e5a_7e1e4fbfccea.slice/crio-937c4ea4212df5034f221069c87e913db818f02ccb1bf39ef5618db72ab02377 WatchSource:0}: Error finding container 937c4ea4212df5034f221069c87e913db818f02ccb1bf39ef5618db72ab02377: Status 404 returned error can't find the container with id 937c4ea4212df5034f221069c87e913db818f02ccb1bf39ef5618db72ab02377 Oct 30 03:02:18.831111 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:18.831005 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:19.034975 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.034942 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:19.035138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.034983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:19.035138 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.035124 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:19.035264 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.035180 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret podName:90b350e9-6998-45b6-882a-17019e89bdf6 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:20.035166994 +0000 UTC m=+3.142013867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret") pod "global-pull-secret-syncer-m266j" (UID: "90b350e9-6998-45b6-882a-17019e89bdf6") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:19.035567 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.035546 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:19.035652 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.035599 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:20.035583979 +0000 UTC m=+3.142430854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:19.136787 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.136121 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:19.136787 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.136307 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:19.136787 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.136328 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:19.136787 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.136341 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gt8qv for pod openshift-network-diagnostics/network-check-target-6szch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:19.136787 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.136398 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv podName:4d22ace0-adef-488a-a025-83b8270d25d9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:20.136381074 +0000 UTC m=+3.243227967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gt8qv" (UniqueName: "kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv") pod "network-check-target-6szch" (UID: "4d22ace0-adef-488a-a025-83b8270d25d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:19.260070 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.260039 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:19.406865 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.406742 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:18 +0000 UTC" deadline="2028-05-02 19:05:06.112315535 +0000 UTC" Oct 30 03:02:19.406865 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.406821 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13216h2m46.705498975s" Oct 30 03:02:19.460062 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.460032 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:19.460217 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:19.460155 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:19.479325 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.479289 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tpl8m" event={"ID":"dde3589d-5a4c-4176-b493-f5ecfa81349f","Type":"ContainerStarted","Data":"216d60ca2d895fec6796ef693433e1acd4f050bfdce4b7a6ec31fc5be06c1489"} Oct 30 03:02:19.489033 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.489001 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shbmk" event={"ID":"50f7ec66-f78a-405d-95c0-47a26e24a69d","Type":"ContainerStarted","Data":"aa0bfb13c8c64fefa115073d60c87b4451f7b8a261dd3a332e37297dd4d42098"} Oct 30 03:02:19.490594 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.490549 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" event={"ID":"454e1d2d-f058-4a2b-a420-298761f2ff6a","Type":"ContainerStarted","Data":"38532d928dadb938ff3c311632a6402097546df91bbcfac7b98ed6cf9dd12754"} Oct 30 03:02:19.495665 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.495635 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerStarted","Data":"6a8bb95189927aa9434203255cc30cacd1a98a6dd47b2bb7d1e54e4831856447"} Oct 30 03:02:19.500653 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.500605 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2q75" event={"ID":"3fcbb594-f806-4ac7-ab99-ea99ca6c827a","Type":"ContainerStarted","Data":"4e6a7846ae8ffb507824ce42381d0dbeb16426ddea22a22dbdff8af67a9a6149"} Oct 30 03:02:19.502816 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.502783 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"937c4ea4212df5034f221069c87e913db818f02ccb1bf39ef5618db72ab02377"} Oct 30 03:02:19.528344 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.528303 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" event={"ID":"655ddd5c-e5dd-4217-b597-6dfb5e33fa73","Type":"ContainerStarted","Data":"bdcb7479e6eb8bd1d6e34340b40cd357bf8f31453dd93f5b5cde5904659aae1d"} Oct 30 03:02:19.540435 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.540402 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hq9jd" event={"ID":"20788c87-fbba-4a1c-9d38-f32396e8ed24","Type":"ContainerStarted","Data":"37803cef26cb335d9c1ba97d96e97cdc36bf69aade2685e53418abdbda7ae7b8"} Oct 30 03:02:19.559352 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.556899 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jj8ks" event={"ID":"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7","Type":"ContainerStarted","Data":"d52adaf7656f9ec4b858a919f8a57863f076a7780a3e974ade6f2d36cfff4f16"} Oct 30 03:02:19.732906 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:19.732786 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:20.044360 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:20.044280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:20.044360 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:20.044317 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:20.044594 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.044454 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:20.044594 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.044513 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret podName:90b350e9-6998-45b6-882a-17019e89bdf6 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:22.044493365 +0000 UTC m=+5.151340240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret") pod "global-pull-secret-syncer-m266j" (UID: "90b350e9-6998-45b6-882a-17019e89bdf6") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:20.044962 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.044928 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:20.045060 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.045013 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:22.044995286 +0000 UTC m=+5.151842179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:20.144925 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:20.144890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:20.145129 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.145109 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:20.145129 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.145128 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:20.145244 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.145140 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gt8qv for pod openshift-network-diagnostics/network-check-target-6szch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:20.145244 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.145195 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv podName:4d22ace0-adef-488a-a025-83b8270d25d9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:22.145176178 +0000 UTC m=+5.252023058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gt8qv" (UniqueName: "kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv") pod "network-check-target-6szch" (UID: "4d22ace0-adef-488a-a025-83b8270d25d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:20.407949 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:20.407803 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:18 +0000 UTC" deadline="2028-07-03 21:15:50.344331234 +0000 UTC" Oct 30 03:02:20.407949 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:20.407872 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14706h13m29.936463243s" Oct 30 03:02:20.457943 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:20.457152 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:20.457943 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.457306 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:20.457943 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:20.457742 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:20.457943 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:20.457854 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:21.024746 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:21.024713 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:21.466654 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:21.466097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:21.466654 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:21.466236 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:22.066228 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.066187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:22.066411 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.066243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:22.066411 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.066392 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:22.066535 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.066463 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret podName:90b350e9-6998-45b6-882a-17019e89bdf6 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:26.066443114 +0000 UTC m=+9.173290010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret") pod "global-pull-secret-syncer-m266j" (UID: "90b350e9-6998-45b6-882a-17019e89bdf6") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:22.066950 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.066895 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:22.066950 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.066950 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:26.066934004 +0000 UTC m=+9.173780894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:22.167389 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.167311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:22.167550 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.167482 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:22.167550 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.167498 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:22.167550 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.167509 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gt8qv for pod openshift-network-diagnostics/network-check-target-6szch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:22.167720 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.167553 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv podName:4d22ace0-adef-488a-a025-83b8270d25d9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:26.167540986 +0000 UTC m=+9.274387860 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gt8qv" (UniqueName: "kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv") pod "network-check-target-6szch" (UID: "4d22ace0-adef-488a-a025-83b8270d25d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:22.458210 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.457713 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:22.458532 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.457732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:22.458532 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.458382 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:22.458532 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:22.458476 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:22.582048 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.581999 2574 generic.go:356] "Generic (PLEG): container finished" podID="dca45c9b1760cdbab09f6b1b711a4a17" containerID="832ec5fda840c2ec4df9faa6923699414cf8bfe16ccfbff7ed32f8d96aa29f03" exitCode=0 Oct 30 03:02:22.582513 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.582101 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" event={"ID":"dca45c9b1760cdbab09f6b1b711a4a17","Type":"ContainerDied","Data":"832ec5fda840c2ec4df9faa6923699414cf8bfe16ccfbff7ed32f8d96aa29f03"} Oct 30 03:02:22.589880 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.589108 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="e5423947d18ded5f1fcaac313b94a6ce61ad30b8758895ce1114692e1b817f79" exitCode=0 Oct 30 03:02:22.589880 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.589193 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerDied","Data":"e5423947d18ded5f1fcaac313b94a6ce61ad30b8758895ce1114692e1b817f79"} Oct 30 03:02:22.595663 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.595215 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" event={"ID":"7c18c0405adfb2995cb6b447bf5eb736","Type":"ContainerStarted","Data":"1982beab2c1b65291da7031eb906596a05bbd7a3275c2f95853b0ec21f8b369e"} Oct 30 03:02:22.595663 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.595440 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:22.617899 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.617407 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal"] Oct 30 03:02:22.617899 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:22.617602 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:23.465023 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:23.462640 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:23.465023 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:23.462846 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:23.608998 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:23.608953 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j2q75" event={"ID":"3fcbb594-f806-4ac7-ab99-ea99ca6c827a","Type":"ContainerStarted","Data":"b786ad59ec773b3000ff0face122fcf935261e7027413fbb447e027d001ea16d"} Oct 30 03:02:23.615536 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:23.615150 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" event={"ID":"dca45c9b1760cdbab09f6b1b711a4a17","Type":"ContainerStarted","Data":"674a113008f39ca43dd97d1b0d3e4e34d57b681f420a8366b5dcc0bbfa6256ff"} Oct 30 03:02:23.653050 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:23.652428 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" podStartSLOduration=1.652408704 podStartE2EDuration="1.652408704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:22.713043599 +0000 UTC m=+5.819890499" watchObservedRunningTime="2026-10-30 03:02:23.652408704 +0000 UTC m=+6.759255604" Oct 30 03:02:23.653050 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:23.652896 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j2q75" podStartSLOduration=1.216899123 podStartE2EDuration="5.65288846s" totalImagesPullingTime="4.435989337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:23.652085546 +0000 UTC m=+6.758932443" watchObservedRunningTime="2026-10-30 03:02:23.65288846 +0000 UTC m=+6.759735362" Oct 30 03:02:23.693723 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:23.693661 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" podStartSLOduration=5.693651029 podStartE2EDuration="5.693651029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:23.693006087 +0000 UTC m=+6.799852986" watchObservedRunningTime="2026-10-30 03:02:23.693651029 +0000 UTC m=+6.800497934" Oct 30 03:02:24.068373 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:24.068339 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:24.069262 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:24.069239 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:24.457405 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:24.457374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:24.457556 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:24.457532 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:24.457637 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:24.457620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:24.457729 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:24.457707 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:24.621785 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:24.621484 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-shbmk" event={"ID":"50f7ec66-f78a-405d-95c0-47a26e24a69d","Type":"ContainerStarted","Data":"49814b3876943c8782c135b4b9741cfd19de046e91bbea4b3d1d84f0d0b342be"} Oct 30 03:02:24.666034 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:24.665787 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-shbmk" podStartSLOduration=1.184298103 podStartE2EDuration="6.665764966s" totalImagesPullingTime="5.481466863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:24.664894626 +0000 UTC m=+7.771741527" watchObservedRunningTime="2026-10-30 03:02:24.665764966 +0000 UTC m=+7.772611866" Oct 30 03:02:25.459621 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:25.459558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:25.459784 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:25.459664 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:25.626016 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:25.625985 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:02:25.626490 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:25.626452 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" event={"ID":"454e1d2d-f058-4a2b-a420-298761f2ff6a","Type":"ContainerStarted","Data":"03a7dbd9cdab8c43edbe3bc8dcb9445f3b906ef9e44537dd3dc3b14931d2f9db"} Oct 30 03:02:26.107852 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:26.107067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:26.107852 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:26.107115 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:26.107852 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.107259 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:26.107852 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.107326 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret podName:90b350e9-6998-45b6-882a-17019e89bdf6 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:34.107305806 +0000 UTC m=+17.214152695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret") pod "global-pull-secret-syncer-m266j" (UID: "90b350e9-6998-45b6-882a-17019e89bdf6") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:26.107852 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.107765 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:26.107852 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.107815 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:34.107801255 +0000 UTC m=+17.214648142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:26.208484 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:26.207823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:26.208484 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.208010 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:26.208484 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.208032 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:26.208484 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.208045 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gt8qv for pod openshift-network-diagnostics/network-check-target-6szch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:26.208484 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.208102 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv podName:4d22ace0-adef-488a-a025-83b8270d25d9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:34.208084578 +0000 UTC m=+17.314931455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gt8qv" (UniqueName: "kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv") pod "network-check-target-6szch" (UID: "4d22ace0-adef-488a-a025-83b8270d25d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:26.457725 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:26.457620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:26.457915 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.457730 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:26.458079 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:26.458038 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:26.458140 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:26.458102 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:26.801700 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:26.801453 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 03:02:27.385455 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.384831 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:02:26.801475555Z","UUID":"56479cee-d30a-4877-8f10-aaf83c5534df","Handler":null,"Name":"","Endpoint":""} Oct 30 03:02:27.390653 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.390629 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 03:02:27.390771 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.390677 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 03:02:27.460998 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.460966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:27.461174 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:27.461077 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:27.637194 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.637152 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jj8ks" event={"ID":"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7","Type":"ContainerStarted","Data":"274b80897bdf3336f4e89e20e2e17f095ab2b873c57d1abb7ad3ec9f1a125613"} Oct 30 03:02:27.639769 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.639728 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tpl8m" event={"ID":"dde3589d-5a4c-4176-b493-f5ecfa81349f","Type":"ContainerStarted","Data":"38e7fe939ae6ab3bcc107cd4bd8929d413c315c60295a349a7778294a6c969c0"} Oct 30 03:02:27.642696 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.642671 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" event={"ID":"454e1d2d-f058-4a2b-a420-298761f2ff6a","Type":"ContainerStarted","Data":"bf95246dc45ae02ffb58eccb2a34e4513795f9245c7086584435a1b7b1ad4852"} Oct 30 03:02:27.645049 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.645022 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="87583a3a207dded8942f5777c333b19bde6aa42be006611f9a3361d6948d3183" exitCode=0 Oct 30 03:02:27.645138 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.645055 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerDied","Data":"87583a3a207dded8942f5777c333b19bde6aa42be006611f9a3361d6948d3183"} Oct 30 03:02:27.725917 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.725157 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jj8ks" podStartSLOduration=2.140493388 podStartE2EDuration="10.725138262s" totalImagesPullingTime="8.584644874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:27.678791063 +0000 UTC m=+10.785637960" watchObservedRunningTime="2026-10-30 03:02:27.725138262 +0000 UTC m=+10.831985161" Oct 30 03:02:27.766443 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:27.766357 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tpl8m" podStartSLOduration=1.292089974 podStartE2EDuration="9.766344286s" totalImagesPullingTime="8.474254312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:27.766098982 +0000 UTC m=+10.872945876" watchObservedRunningTime="2026-10-30 03:02:27.766344286 +0000 UTC m=+10.873191180" Oct 30 03:02:28.458243 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:28.458024 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:28.458556 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:28.458353 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:28.458656 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:28.458071 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:28.458787 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:28.458747 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:28.649719 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:28.649631 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" event={"ID":"454e1d2d-f058-4a2b-a420-298761f2ff6a","Type":"ContainerStarted","Data":"41354679f892c826d1b3d9e71383f3c96117728302bee50435366464b54f3fb2"} Oct 30 03:02:28.652939 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:28.652903 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="5c2399eaec814d0bcf816faff7326e1a3363c3bdd5e7bf542647b805f147ce59" exitCode=0 Oct 30 03:02:28.653716 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:28.653685 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerDied","Data":"5c2399eaec814d0bcf816faff7326e1a3363c3bdd5e7bf542647b805f147ce59"} Oct 30 03:02:28.692894 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:28.692840 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" podStartSLOduration=2.830502505 podStartE2EDuration="11.692822445s" totalImagesPullingTime="8.86231994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:28.692186425 +0000 UTC m=+11.799033321" watchObservedRunningTime="2026-10-30 03:02:28.692822445 +0000 UTC m=+11.799669343" Oct 30 03:02:29.457437 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:29.457407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:29.457610 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:29.457537 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:29.656251 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:29.656219 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hq9jd" event={"ID":"20788c87-fbba-4a1c-9d38-f32396e8ed24","Type":"ContainerStarted","Data":"995facec0994c8b3cb7670d4a380450770549eb6407c21c8fbf99836010c1bb3"} Oct 30 03:02:29.696633 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:29.696591 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hq9jd" podStartSLOduration=4.134797608 podStartE2EDuration="12.696577159s" totalImagesPullingTime="8.561779551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:29.696369511 +0000 UTC m=+12.803216406" watchObservedRunningTime="2026-10-30 03:02:29.696577159 +0000 UTC m=+12.803424053" Oct 30 03:02:30.457687 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:30.457655 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:30.457893 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:30.457767 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:30.457893 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:30.457810 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:30.458080 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:30.457943 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:30.659879 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:30.659532 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" event={"ID":"655ddd5c-e5dd-4217-b597-6dfb5e33fa73","Type":"ContainerStarted","Data":"878e0b7793dc9e6686e89ee623319853d88a9e926d970e0cd9a89c29c3221d7c"} Oct 30 03:02:30.662161 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:30.662135 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="722d736b88beaf4b63cf75b788a0b21235a4a5baee97c098dd87bb529efd52c9" exitCode=0 Oct 30 03:02:30.662289 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:30.662227 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerDied","Data":"722d736b88beaf4b63cf75b788a0b21235a4a5baee97c098dd87bb529efd52c9"} Oct 30 03:02:30.705408 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:30.705365 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" podStartSLOduration=2.798877402 podStartE2EDuration="13.705347878s" totalImagesPullingTime="10.906470476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:30.704769354 +0000 UTC m=+13.811616251" watchObservedRunningTime="2026-10-30 03:02:30.705347878 +0000 UTC m=+13.812194774" Oct 30 03:02:31.458008 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:31.457975 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:31.458191 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:31.458110 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:32.457126 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:32.457098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:32.457673 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:32.457254 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:32.457673 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:32.457294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:32.457673 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:32.457396 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:32.670355 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:32.670080 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="0fcd1e8fc72ad168c1860893775cf1c064c7e6b0942b3ce0ee3f675ee416ef9a" exitCode=0 Oct 30 03:02:32.670355 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:32.670167 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerDied","Data":"0fcd1e8fc72ad168c1860893775cf1c064c7e6b0942b3ce0ee3f675ee416ef9a"} Oct 30 03:02:33.457448 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:33.457415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:33.457832 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:33.457539 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:33.675701 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:33.675665 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="cb4add2f0709a4d315d6dc7f39089676ed4d3606ea83ccc82a0e27ab7cc46309" exitCode=0 Oct 30 03:02:33.675858 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:33.675715 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerDied","Data":"cb4add2f0709a4d315d6dc7f39089676ed4d3606ea83ccc82a0e27ab7cc46309"} Oct 30 03:02:34.165925 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.165885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:34.165925 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.165921 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:34.166126 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.166026 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:34.166126 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.166029 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:34.166126 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.166077 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret podName:90b350e9-6998-45b6-882a-17019e89bdf6 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.166063624 +0000 UTC m=+33.272910518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret") pod "global-pull-secret-syncer-m266j" (UID: "90b350e9-6998-45b6-882a-17019e89bdf6") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:02:34.166126 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.166090 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.166084546 +0000 UTC m=+33.272931418 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:02:34.266623 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.266402 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:34.266623 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.266568 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:02:34.266623 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.266590 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:02:34.266623 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.266600 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gt8qv for pod openshift-network-diagnostics/network-check-target-6szch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:34.266871 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.266649 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv podName:4d22ace0-adef-488a-a025-83b8270d25d9 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.266636376 +0000 UTC m=+33.373483262 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gt8qv" (UniqueName: "kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv") pod "network-check-target-6szch" (UID: "4d22ace0-adef-488a-a025-83b8270d25d9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:02:34.458019 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.457992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:34.458401 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.458094 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:34.458401 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.458147 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:34.458401 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:34.458228 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:34.681336 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.681251 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" event={"ID":"9e9460d2-8d0f-4aca-a2a0-860137f1b801","Type":"ContainerStarted","Data":"1a5e89f64062463cd871bdf9005a0fe7bdf8fcd2e956878cc3a37f28b6c6400b"} Oct 30 03:02:34.683839 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.683820 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:02:34.684171 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.684146 2574 generic.go:356] "Generic (PLEG): container finished" podID="585acb34-e959-4d7b-9e5a-7e1e4fbfccea" containerID="5002bbb2b237ed1479172b4defee2ff013d8305ffcdd720b8b978e3d848a6c2a" exitCode=1 Oct 30 03:02:34.684216 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.684191 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"fb1f47890182f59e2bd3f065fbbd9277a9132575116304dfb56935e5117fa0fa"} Oct 30 03:02:34.684281 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.684270 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"88ad512529c64ec35fcdd1b939ed7ad9222581b6b15b15496d369bfecdb6388b"} Oct 30 03:02:34.684311 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.684305 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"922aa1ed324ec96eb3ad149defef997c9e7aba2d1a0b027c14eb6b801e345b05"} Oct 30 03:02:34.684341 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.684333 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"6e43e680a254ab00d6fcd15edbae3bfbb93331f1372e2bcdf8c9da3a76edfc26"} Oct 30 03:02:34.684371 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.684361 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerDied","Data":"5002bbb2b237ed1479172b4defee2ff013d8305ffcdd720b8b978e3d848a6c2a"} Oct 30 03:02:34.684401 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.684391 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"e5200bb40da538bb7a958da7b7a58ddf152e07da10f19d1616ae19d72355fe76"} Oct 30 03:02:34.726202 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:34.726169 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9fhjj" podStartSLOduration=6.762697816 podStartE2EDuration="17.726158428s" totalImagesPullingTime="10.963460612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:34.725889655 +0000 UTC m=+17.832736551" watchObservedRunningTime="2026-10-30 03:02:34.726158428 +0000 UTC m=+17.833005322" Oct 30 03:02:35.457277 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:35.457247 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:35.457437 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:35.457346 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:36.457588 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:36.457513 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:36.457955 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:36.457613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:36.457955 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:36.457668 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:36.457955 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:36.457772 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:36.691036 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:36.691010 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:02:36.691386 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:36.691364 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"dbc810d55aa57a2b4ff10dcab48832a7d29a9acf3ee955e30663d1011821aedd"} Oct 30 03:02:37.458171 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:37.458145 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:37.458613 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:37.458221 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:38.141791 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.141741 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:38.141995 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.141909 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:02:38.142335 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.142313 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:38.457382 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.457310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:38.457517 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:38.457419 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:38.457517 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.457469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:38.457596 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:38.457568 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:38.698084 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.698057 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:02:38.698870 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.698438 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"ce9910c8687a737c152ec87eb8aac532a134a28ff6526e0fda5467b16279e389"} Oct 30 03:02:38.698870 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:38.698860 2574 scope.go:122] "RemoveContainer" containerID="5002bbb2b237ed1479172b4defee2ff013d8305ffcdd720b8b978e3d848a6c2a" Oct 30 03:02:39.457278 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.457243 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:39.457545 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:39.457498 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:39.703544 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.703517 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:02:39.703966 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.703854 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" event={"ID":"585acb34-e959-4d7b-9e5a-7e1e4fbfccea","Type":"ContainerStarted","Data":"6ea943ddde0feb2ea48facf572627e2477428cf76be79212bea14b257898d104"} Oct 30 03:02:39.704145 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.704129 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:39.704269 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.704253 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:39.704411 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.704301 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:39.720951 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.720891 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:39.721061 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.720990 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:39.793955 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:39.793907 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" podStartSLOduration=7.851636746 podStartE2EDuration="22.793892847s" totalImagesPullingTime="14.942256101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:39.793512216 +0000 UTC m=+22.900359123" watchObservedRunningTime="2026-10-30 03:02:39.793892847 +0000 UTC m=+22.900739741" Oct 30 03:02:40.359563 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:40.359216 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6szch"] Oct 30 03:02:40.359563 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:40.359478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:40.359922 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:40.359567 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:40.361910 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:40.361887 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qxx87"] Oct 30 03:02:40.362018 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:40.361987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:40.362085 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:40.362070 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:40.362361 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:40.362344 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m266j"] Oct 30 03:02:40.362433 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:40.362423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:40.362505 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:40.362490 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:41.075647 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:41.075612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:41.206884 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:41.206853 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:41.308309 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:41.308278 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:41.393870 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:41.393787 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:41.457121 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:41.457090 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:41.457272 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:41.457186 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:42.457539 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:42.457504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:42.457539 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:42.457537 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:42.457945 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:42.457612 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:42.457945 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:42.457721 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:43.458129 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:43.458101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:43.458495 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:43.458194 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:44.457363 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:44.457331 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:44.457552 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:44.457450 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qxx87" podUID="cc68d015-081c-41bc-b7ec-2443b5693c76" Oct 30 03:02:44.457552 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:44.457501 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:44.457656 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:44.457587 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m266j" podUID="90b350e9-6998-45b6-882a-17019e89bdf6" Oct 30 03:02:44.697050 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:44.696968 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:45.457853 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:45.457799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:45.458068 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:45.457938 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6szch" podUID="4d22ace0-adef-488a-a025-83b8270d25d9" Oct 30 03:02:46.171503 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.171475 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeReady" Oct 30 03:02:46.171914 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.171569 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 03:02:46.241186 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.241120 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pmfcp"] Oct 30 03:02:46.244202 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.244186 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-9625v"] Oct 30 03:02:46.244320 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.244302 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.247082 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.247064 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.262331 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.262308 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmfcp"] Oct 30 03:02:46.264950 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.264929 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9625v"] Oct 30 03:02:46.275194 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.275176 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 03:02:46.275194 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.275182 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 03:02:46.277665 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.277647 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 03:02:46.277665 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.277664 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 03:02:46.277802 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.277680 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgj97\"" Oct 30 03:02:46.277802 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.277682 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 03:02:46.277802 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.277699 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vnb2g\"" Oct 30 03:02:46.364080 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.364055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qn9q\" (UniqueName: \"kubernetes.io/projected/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-kube-api-access-7qn9q\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.364181 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.364098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-config-volume\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.364181 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.364123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.364248 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.364226 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bd5e4c-0976-4383-82f7-2e8dcb385115-cert\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.364280 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.364253 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnqv9\" (UniqueName: \"kubernetes.io/projected/19bd5e4c-0976-4383-82f7-2e8dcb385115-kube-api-access-bnqv9\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.364311 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.364279 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-tmp-dir\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.457069 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.457046 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:46.457150 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.457076 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:46.464678 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.464657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-config-volume\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.464813 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.464699 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.464813 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.464728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bd5e4c-0976-4383-82f7-2e8dcb385115-cert\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.464813 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.464767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqv9\" (UniqueName: \"kubernetes.io/projected/19bd5e4c-0976-4383-82f7-2e8dcb385115-kube-api-access-bnqv9\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.464813 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.464799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-tmp-dir\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.465017 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.464836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qn9q\" (UniqueName: \"kubernetes.io/projected/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-kube-api-access-7qn9q\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.465017 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:46.464839 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:46.465017 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:46.464974 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:46.964955298 +0000 UTC m=+30.071802188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:46.465149 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.465120 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-tmp-dir\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.465370 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.465344 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-config-volume\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.468617 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.468594 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bd5e4c-0976-4383-82f7-2e8dcb385115-cert\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.473339 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.473319 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnh6p\"" Oct 30 03:02:46.473401 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.473345 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 03:02:46.473401 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.473386 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-69wt9\"" Oct 30 03:02:46.473474 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.473409 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 03:02:46.485527 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.485503 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qn9q\" (UniqueName: \"kubernetes.io/projected/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-kube-api-access-7qn9q\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.485744 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.485728 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqv9\" (UniqueName: \"kubernetes.io/projected/19bd5e4c-0976-4383-82f7-2e8dcb385115-kube-api-access-bnqv9\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.556314 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.556262 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:46.763277 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.763245 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:46.867164 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.867088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:46.872543 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.872522 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pmfcp"] Oct 30 03:02:46.874728 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:46.874704 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bd5e4c_0976_4383_82f7_2e8dcb385115.slice/crio-16671a56f4f848ac148a07362b8ba6c1145934bb57c330abcff45e8f1c78cd6e WatchSource:0}: Error finding container 16671a56f4f848ac148a07362b8ba6c1145934bb57c330abcff45e8f1c78cd6e: Status 404 returned error can't find the container with id 16671a56f4f848ac148a07362b8ba6c1145934bb57c330abcff45e8f1c78cd6e Oct 30 03:02:46.901114 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.901089 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cp2d8"] Oct 30 03:02:46.905188 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.905174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:46.926997 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.926972 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 03:02:46.931275 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.931244 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qj6bv\"" Oct 30 03:02:46.932934 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.932903 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 03:02:46.933150 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.933130 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 03:02:46.933221 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.933187 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 03:02:46.933390 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.933375 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cp2d8"] Oct 30 03:02:46.968730 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.968702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:46.968848 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:46.968834 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:46.968901 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:46.968892 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:47.968876298 +0000 UTC m=+31.075723172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:46.973979 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:46.973958 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:47.069187 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.069158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.069283 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.069189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88zm\" (UniqueName: \"kubernetes.io/projected/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-api-access-h88zm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.069283 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.069212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.069351 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.069340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c3f2dca-e9db-4210-b534-527ed98e7333-crio-socket\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.069381 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.069357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f2dca-e9db-4210-b534-527ed98e7333-data-volume\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.170243 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.170179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c3f2dca-e9db-4210-b534-527ed98e7333-crio-socket\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.170243 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.170206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f2dca-e9db-4210-b534-527ed98e7333-data-volume\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.170243 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.170241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.170445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.170257 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88zm\" (UniqueName: \"kubernetes.io/projected/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-api-access-h88zm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.170445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.170280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.170445 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.170350 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:47.170445 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.170358 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.170445 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.170408 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c3f2dca-e9db-4210-b534-527ed98e7333-crio-socket\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.170776 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.170736 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:47.670403396 +0000 UTC m=+30.777250288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:47.170878 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.170862 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:47.670772244 +0000 UTC m=+30.777619121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.171002 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.170980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f2dca-e9db-4210-b534-527ed98e7333-data-volume\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.192357 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.192334 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88zm\" (UniqueName: \"kubernetes.io/projected/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-api-access-h88zm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.458113 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.458048 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:47.474183 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.474147 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 03:02:47.474318 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.474302 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 03:02:47.474444 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.474428 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-shnp4\"" Oct 30 03:02:47.674063 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.674029 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.674231 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.674068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:47.674231 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.674155 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:47.674231 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.674182 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.674231 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.674223 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.674205403 +0000 UTC m=+31.781052280 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:47.674446 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.674241 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:48.674231919 +0000 UTC m=+31.781078792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:47.683496 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.683454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:47.722805 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.722713 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmfcp" event={"ID":"19bd5e4c-0976-4383-82f7-2e8dcb385115","Type":"ContainerStarted","Data":"16671a56f4f848ac148a07362b8ba6c1145934bb57c330abcff45e8f1c78cd6e"} Oct 30 03:02:47.750132 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.750101 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx"] Oct 30 03:02:47.781387 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.781363 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx"] Oct 30 03:02:47.781518 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.781472 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" Oct 30 03:02:47.783630 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.783606 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:47.802126 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.802098 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hfhbp\"" Oct 30 03:02:47.876280 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.876243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slk6p\" (UniqueName: \"kubernetes.io/projected/8434e5c6-4079-46cc-84d5-6672cbac7ad8-kube-api-access-slk6p\") pod \"migrator-66df5d496b-fn9bx\" (UID: \"8434e5c6-4079-46cc-84d5-6672cbac7ad8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" Oct 30 03:02:47.884035 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.883995 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:47.893670 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.893643 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 03:02:47.893670 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.893643 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:47.962836 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.962801 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:47.976785 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.976691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slk6p\" (UniqueName: \"kubernetes.io/projected/8434e5c6-4079-46cc-84d5-6672cbac7ad8-kube-api-access-slk6p\") pod \"migrator-66df5d496b-fn9bx\" (UID: \"8434e5c6-4079-46cc-84d5-6672cbac7ad8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" Oct 30 03:02:47.976785 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.976770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:47.976960 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.976854 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:47.976960 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:47.976905 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:49.97688805 +0000 UTC m=+33.083734941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:47.997461 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:47.997433 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slk6p\" (UniqueName: \"kubernetes.io/projected/8434e5c6-4079-46cc-84d5-6672cbac7ad8-kube-api-access-slk6p\") pod \"migrator-66df5d496b-fn9bx\" (UID: \"8434e5c6-4079-46cc-84d5-6672cbac7ad8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" Oct 30 03:02:48.106483 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:48.106455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" Oct 30 03:02:48.207171 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:48.207136 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:48.306555 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:48.306469 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:48.434543 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:48.434515 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx"] Oct 30 03:02:48.436451 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:48.436417 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8434e5c6_4079_46cc_84d5_6672cbac7ad8.slice/crio-30bdbb82a80a556514769a0aa22e1e07cd30f4746d96ed9132351047d4c9f3e3 WatchSource:0}: Error finding container 30bdbb82a80a556514769a0aa22e1e07cd30f4746d96ed9132351047d4c9f3e3: Status 404 returned error can't find the container with id 30bdbb82a80a556514769a0aa22e1e07cd30f4746d96ed9132351047d4c9f3e3 Oct 30 03:02:48.682060 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:48.681975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:48.682060 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:48.682021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:48.682252 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:48.682107 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:48.682252 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:48.682150 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:48.682252 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:48.682178 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.682158538 +0000 UTC m=+33.789005434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:48.682252 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:48.682213 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:50.682195934 +0000 UTC m=+33.789042821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:48.726381 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:48.726340 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" event={"ID":"8434e5c6-4079-46cc-84d5-6672cbac7ad8","Type":"ContainerStarted","Data":"30bdbb82a80a556514769a0aa22e1e07cd30f4746d96ed9132351047d4c9f3e3"} Oct 30 03:02:49.024229 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:49.024151 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:49.128250 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:49.128212 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:49.578328 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:49.578294 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:49.993772 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:49.993724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:49.994007 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:49.993869 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:49.994007 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:49.993923 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:53.993909549 +0000 UTC m=+37.100756422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:50.195488 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.195422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:50.195488 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.195456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:50.195637 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:50.195551 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:50.195637 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:50.195613 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:22.195596182 +0000 UTC m=+65.302443081 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : secret "metrics-daemon-secret" not found Oct 30 03:02:50.197719 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.197701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:50.295936 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.295905 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:50.298649 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.298619 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:50.377176 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.377149 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:50.473346 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.473319 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:50.697463 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.697439 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m266j"] Oct 30 03:02:50.697728 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.697631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:50.697728 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.697659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:50.697825 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:50.697777 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:50.697825 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:50.697780 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:50.697886 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:50.697841 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.697828433 +0000 UTC m=+37.804675306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:50.697886 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:50.697857 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:54.697850931 +0000 UTC m=+37.804697804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:50.699008 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:50.698986 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b350e9_6998_45b6_882a_17019e89bdf6.slice/crio-c55b339dc474fdf07b3cf868a5755a272fbaba0d871a270328c12bdc620b687f WatchSource:0}: Error finding container c55b339dc474fdf07b3cf868a5755a272fbaba0d871a270328c12bdc620b687f: Status 404 returned error can't find the container with id c55b339dc474fdf07b3cf868a5755a272fbaba0d871a270328c12bdc620b687f Oct 30 03:02:50.747131 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.747090 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m266j" event={"ID":"90b350e9-6998-45b6-882a-17019e89bdf6","Type":"ContainerStarted","Data":"c55b339dc474fdf07b3cf868a5755a272fbaba0d871a270328c12bdc620b687f"} Oct 30 03:02:50.748595 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.748569 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pmfcp" event={"ID":"19bd5e4c-0976-4383-82f7-2e8dcb385115","Type":"ContainerStarted","Data":"d191aa0b479a72b6c15bbe75e5132f79d5c359be32ee2064dc48a1328ef76b1b"} Oct 30 03:02:50.750030 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.750010 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" event={"ID":"8434e5c6-4079-46cc-84d5-6672cbac7ad8","Type":"ContainerStarted","Data":"2ba08ba3dcb6856f2982089400371ce80042695de140e82ca91c3f759121fa70"} Oct 30 03:02:50.776997 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.776978 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6szch"] Oct 30 03:02:50.777982 ip-10-0-134-92 kubenswrapper[2574]: W1030 03:02:50.777964 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d22ace0_adef_488a_a025_83b8270d25d9.slice/crio-82bd5ffa790f687c2b2b9d7a7c0939d5a48a02ef705b108160c0462ac27d0932 WatchSource:0}: Error finding container 82bd5ffa790f687c2b2b9d7a7c0939d5a48a02ef705b108160c0462ac27d0932: Status 404 returned error can't find the container with id 82bd5ffa790f687c2b2b9d7a7c0939d5a48a02ef705b108160c0462ac27d0932 Oct 30 03:02:50.788962 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.788927 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pmfcp" podStartSLOduration=1.9620485410000001 podStartE2EDuration="4.788916898s" totalImagesPullingTime="2.826868357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:50.788443398 +0000 UTC m=+33.895290293" watchObservedRunningTime="2026-10-30 03:02:50.788916898 +0000 UTC m=+33.895763796" Oct 30 03:02:50.830070 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:50.830031 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" podStartSLOduration=2.463027186 podStartE2EDuration="3.830020015s" totalImagesPullingTime="1.366992829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:50.829249547 +0000 UTC m=+33.936096441" watchObservedRunningTime="2026-10-30 03:02:50.830020015 +0000 UTC m=+33.936866907" Oct 30 03:02:51.410788 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:51.410733 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:51.511832 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:51.511746 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:51.754720 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:51.754634 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6szch" event={"ID":"4d22ace0-adef-488a-a025-83b8270d25d9","Type":"ContainerStarted","Data":"82bd5ffa790f687c2b2b9d7a7c0939d5a48a02ef705b108160c0462ac27d0932"} Oct 30 03:02:52.506715 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:52.506684 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:54.022438 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.022404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:54.022849 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:54.022583 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:54.022849 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:54.022662 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:02.022641471 +0000 UTC m=+45.129488359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:54.106522 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.106492 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jj8ks_436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7/dns-node-resolver/0.log" Oct 30 03:02:54.726581 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.726545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:54.726581 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.726595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:54.726881 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:54.726705 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:54.726881 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:54.726710 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:54.726881 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:54.726789 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:02.7267691 +0000 UTC m=+45.833615987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:54.726881 ip-10-0-134-92 kubenswrapper[2574]: E1030 03:02:54.726806 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:02.726797917 +0000 UTC m=+45.833644790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:54.764407 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.764370 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6szch" event={"ID":"4d22ace0-adef-488a-a025-83b8270d25d9","Type":"ContainerStarted","Data":"0cc5ac82c6cbaa733b473f54f6ac8de6d6583c48d1abbca7a1f8668e032990af"} Oct 30 03:02:54.764605 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.764586 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:54.808036 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.807976 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6szch" podStartSLOduration=34.415115116 podStartE2EDuration="37.807959801s" totalImagesPullingTime="3.392844685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:02:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:54.80754306 +0000 UTC m=+37.914389956" watchObservedRunningTime="2026-10-30 03:02:54.807959801 +0000 UTC m=+37.914806700" Oct 30 03:02:54.867082 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.867041 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:54.906315 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:54.906291 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shbmk_50f7ec66-f78a-405d-95c0-47a26e24a69d/node-ca/0.log" Oct 30 03:02:55.006778 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.006672 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:55.122468 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.122430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:55.214804 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.214765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:55.506116 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.506013 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pmfcp_19bd5e4c-0976-4383-82f7-2e8dcb385115/serve-healthcheck-canary/0.log" Oct 30 03:02:55.702147 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.702120 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:55.768568 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.768492 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m266j" event={"ID":"90b350e9-6998-45b6-882a-17019e89bdf6","Type":"ContainerStarted","Data":"bd60c7a71031d381e6edabf777f78f7bf365d2b635170cd5dd9cc4824214922c"} Oct 30 03:02:55.808200 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.808149 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m266j" podStartSLOduration=32.881102532 podStartE2EDuration="37.80813742s" totalImagesPullingTime="4.927034888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:55.807892587 +0000 UTC m=+38.914739481" watchObservedRunningTime="2026-10-30 03:02:55.80813742 +0000 UTC m=+38.914984314" Oct 30 03:02:55.933556 ip-10-0-134-92 kubenswrapper[2574]: I1030 03:02:55.933489 2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:55.933638 ip-10-0-134-92 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 03:02:55.958298 ip-10-0-134-92 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 03:02:55.958474 ip-10-0-134-92 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 03:02:55.958520 ip-10-0-134-92 systemd[1]: kubelet.service: Consumed 2.253s CPU time. Oct 30 03:02:55.968252 ip-10-0-134-92 systemd[1]: Starting Kubernetes Kubelet... Oct 30 03:02:56.026149 ip-10-0-134-92 kubenswrapper[4442]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:56.026149 ip-10-0-134-92 kubenswrapper[4442]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 03:02:56.026149 ip-10-0-134-92 kubenswrapper[4442]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:56.026149 ip-10-0-134-92 kubenswrapper[4442]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:02:56.028136 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028064 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.028136 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028137 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028141 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028145 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028148 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028152 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028154 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028157 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028160 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028162 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028166 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028171 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028175 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028178 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028181 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028184 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028187 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028189 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028192 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028195 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.028205 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028197 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028200 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028203 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028206 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028208 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028211 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028214 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028220 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028223 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028225 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028228 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028231 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028233 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028236 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028239 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028241 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028246 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028250 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028253 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.028700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028255 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028258 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028260 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028263 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028265 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028268 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028270 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028273 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028281 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028284 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028286 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028289 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028293 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028295 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028298 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028304 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028307 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028310 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028312 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028315 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.029188 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028317 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028320 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028322 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028325 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028328 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028330 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028333 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028335 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028338 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028340 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028342 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028345 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028347 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028350 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028352 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028355 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028357 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028359 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028362 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028365 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.029665 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028367 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028370 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028373 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028375 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028377 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028380 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028382 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028386 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028389 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028391 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028394 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028397 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028399 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028402 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028404 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028407 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028410 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028413 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028415 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028417 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.030178 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028420 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028422 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028425 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028427 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028430 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028432 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.028434 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029224 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029231 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029234 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029236 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029239 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029242 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029244 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029247 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029250 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029253 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029256 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029258 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.030648 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029262 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029264 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029267 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029269 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029272 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029275 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029277 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029280 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029282 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029285 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029287 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029290 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029293 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029295 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029298 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029300 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029303 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029305 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029308 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029310 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.031324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029313 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029315 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029318 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029320 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029323 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029325 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029328 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029330 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029333 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029336 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029339 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029341 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029345 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029347 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029350 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029352 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029355 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029357 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029360 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.031857 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029362 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029365 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029367 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029370 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029372 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029375 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029378 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029380 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029383 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029385 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029387 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029390 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029392 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029395 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029397 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029399 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029404 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029407 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029410 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029413 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.032324 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029416 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029418 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029421 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029424 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029426 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029429 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029431 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029434 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029436 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029439 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029441 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029444 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029446 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029449 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029451 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029454 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029456 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029459 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029461 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029464 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.032872 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029466 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029469 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029472 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029474 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029478 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029481 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029484 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029487 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029489 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029492 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029494 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029497 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029500 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029502 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.029505 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030121 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030128 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030131 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030135 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.033353 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030137 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030140 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030143 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030146 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030148 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030151 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030156 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030158 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030161 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030163 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030166 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030168 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030171 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030173 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030176 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030178 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030181 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030183 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030186 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.033862 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030189 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030191 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030194 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030197 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030199 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030204 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030207 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030210 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030212 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030221 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030223 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030226 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030228 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030231 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030234 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030236 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030239 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030241 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030244 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.034328 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030247 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030250 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030254 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030257 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030259 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030262 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030264 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030267 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030269 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030272 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030274 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030276 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030279 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030281 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030284 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030288 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030291 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030294 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030296 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030299 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.034803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030301 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030304 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030306 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030315 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030318 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030321 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030323 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030326 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030328 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030331 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030333 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030336 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030339 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030343 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030346 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030349 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030351 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030354 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030356 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030359 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.035289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030361 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030364 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030367 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030369 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030372 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030374 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030376 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030379 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030381 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030384 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030386 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030389 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030391 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030394 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030396 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030398 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030406 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030409 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030411 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030414 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.035787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030416 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030418 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030421 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.030424 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030533 4442 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030540 4442 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030550 4442 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030554 4442 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030559 4442 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030562 4442 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030566 4442 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030570 4442 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030573 4442 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030576 4442 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030580 4442 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030583 4442 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030586 4442 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030589 4442 flags.go:64] FLAG: --cgroup-root="" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030592 4442 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030595 4442 flags.go:64] FLAG: --client-ca-file="" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030598 4442 flags.go:64] FLAG: --cloud-provider="external" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030600 4442 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030608 4442 flags.go:64] FLAG: --cluster-domain="" Oct 30 03:02:56.036276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030610 4442 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030617 4442 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030620 4442 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030624 4442 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030627 4442 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030630 4442 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030638 4442 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030642 4442 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030645 4442 flags.go:64] FLAG: --contention-profiling="false" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030648 4442 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030651 4442 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030654 4442 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030657 4442 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030663 4442 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030666 4442 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030670 4442 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030673 4442 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030676 4442 flags.go:64] FLAG: --enable-server="true" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030679 4442 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030685 4442 flags.go:64] FLAG: --event-burst="100" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030688 4442 flags.go:64] FLAG: --event-qps="50" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030691 4442 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030694 4442 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030697 4442 flags.go:64] FLAG: --eviction-hard="" Oct 30 03:02:56.036911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030701 4442 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030704 4442 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030707 4442 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030710 4442 flags.go:64] FLAG: --eviction-soft="" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030713 4442 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030716 4442 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030719 4442 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030722 4442 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030725 4442 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030727 4442 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030730 4442 flags.go:64] FLAG: --feature-gates="" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030734 4442 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030737 4442 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030740 4442 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030743 4442 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030768 4442 flags.go:64] FLAG: --healthz-port="10248" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030772 4442 flags.go:64] FLAG: --help="false" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030775 4442 flags.go:64] FLAG: --hostname-override="ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030778 4442 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030781 4442 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030784 4442 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030789 4442 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030792 4442 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030796 4442 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 03:02:56.037984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030799 4442 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030802 4442 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030805 4442 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030808 4442 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030811 4442 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030814 4442 flags.go:64] FLAG: --kube-reserved="" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030817 4442 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030820 4442 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030823 4442 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030825 4442 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030828 4442 flags.go:64] FLAG: --lock-file="" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030831 4442 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030834 4442 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030837 4442 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030842 4442 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030845 4442 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030848 4442 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030850 4442 flags.go:64] FLAG: --logging-format="text" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030853 4442 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030857 4442 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030859 4442 flags.go:64] FLAG: --manifest-url="" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030862 4442 flags.go:64] FLAG: --manifest-url-header="" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030867 4442 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030870 4442 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030879 4442 flags.go:64] FLAG: --max-pods="110" Oct 30 03:02:56.038565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030882 4442 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030885 4442 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030888 4442 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030891 4442 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030895 4442 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030898 4442 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030902 4442 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030909 4442 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030912 4442 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030915 4442 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030918 4442 flags.go:64] FLAG: --pod-cidr="" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030921 4442 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030923 4442 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030926 4442 flags.go:64] FLAG: --pods-per-core="0" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030929 4442 flags.go:64] FLAG: --port="10250" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030932 4442 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030935 4442 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02f043e8c8fec38fc" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030938 4442 flags.go:64] FLAG: --qos-reserved="" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030941 4442 flags.go:64] FLAG: --read-only-port="10255" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030944 4442 flags.go:64] FLAG: --register-node="true" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030946 4442 flags.go:64] FLAG: --register-with-taints="" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030950 4442 flags.go:64] FLAG: --registry-burst="10" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030953 4442 flags.go:64] FLAG: --registry-qps="5" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030955 4442 flags.go:64] FLAG: --reserved-cpus="" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030958 4442 flags.go:64] FLAG: --reserved-memory="" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030962 4442 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 03:02:56.039208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030965 4442 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030968 4442 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030971 4442 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030973 4442 flags.go:64] FLAG: --runonce="false" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030976 4442 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030979 4442 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030988 4442 flags.go:64] FLAG: --seccomp-default="false" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030991 4442 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030994 4442 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.030997 4442 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031001 4442 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031004 4442 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031008 4442 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031011 4442 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031014 4442 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031016 4442 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031019 4442 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031022 4442 flags.go:64] FLAG: --system-cgroups="" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031025 4442 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031036 4442 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031039 4442 flags.go:64] FLAG: --tls-cert-file="" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031041 4442 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031048 4442 flags.go:64] FLAG: --tls-min-version="" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031051 4442 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031054 4442 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 03:02:56.039863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031057 4442 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031059 4442 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031062 4442 flags.go:64] FLAG: --v="2" Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031066 4442 flags.go:64] FLAG: --version="false" Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031070 4442 flags.go:64] FLAG: --vmodule="" Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031074 4442 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031077 4442 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031263 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031267 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031270 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031273 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031276 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031278 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031281 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031289 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031291 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031294 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031298 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031301 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031305 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031307 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.040461 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031310 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031312 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031315 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031317 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031320 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031322 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031326 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031328 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031331 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031334 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031336 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031339 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031342 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031344 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031347 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031349 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031352 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031354 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031357 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031359 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.041048 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031362 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031364 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031367 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031369 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031372 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031374 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031382 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031385 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031389 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031392 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031396 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031398 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031401 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031404 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031406 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031409 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031412 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031414 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031417 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.041526 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031421 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031425 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031428 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031431 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031434 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031437 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031439 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031442 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031445 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031449 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031452 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031455 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031458 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031461 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031463 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031466 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031469 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031471 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031474 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.042128 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031477 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031485 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031489 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031491 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031495 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031498 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031500 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031503 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031506 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031508 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031511 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031513 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031516 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031518 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031521 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031523 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031526 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031528 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031531 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.042655 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031533 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031536 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031539 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031541 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031544 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031546 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031548 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031551 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031554 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031556 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031559 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031561 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031563 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031566 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031569 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.031600 4442 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:56.043190 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031641 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031646 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031649 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031652 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031655 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031658 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031661 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031664 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031666 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031668 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031671 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031673 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031676 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031678 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031681 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031683 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031686 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031689 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031691 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031694 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.043628 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031696 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031698 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031701 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031703 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031706 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031708 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031711 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031713 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031717 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031721 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031724 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031727 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031737 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031741 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031743 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031746 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031761 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031765 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031770 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.044218 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031773 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031776 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031779 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031781 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031784 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031786 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031790 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031792 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031795 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031797 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031800 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031802 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031805 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031807 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031809 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031812 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031814 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031817 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031819 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.044833 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031822 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031824 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031827 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031829 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031832 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031834 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031839 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031850 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031853 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031855 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031858 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031860 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031863 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031866 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031869 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031871 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031874 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031876 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031879 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031881 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.045294 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031884 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031886 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031888 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031891 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031893 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031896 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031898 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031901 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031903 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031906 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031908 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031911 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031913 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031915 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031918 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031920 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031922 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031925 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031928 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.045986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031932 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031940 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031943 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031946 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031949 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031952 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031954 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031957 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.031959 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.035821 4442 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.035836 4442 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.035852 4442 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.035856 4442 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035914 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035919 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035922 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035925 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035928 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035930 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.046514 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035933 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035935 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035937 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035941 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035943 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035946 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035948 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035951 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035954 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035956 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035959 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035961 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035964 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035968 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035971 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035974 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035977 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035979 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035981 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035984 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.047026 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035986 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035989 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035991 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035994 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.035997 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036001 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036004 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036006 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036009 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036012 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036014 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036017 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036019 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036022 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036024 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036027 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036030 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036033 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036037 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.047494 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036040 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036042 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036045 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036048 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036052 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036055 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036058 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036060 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036063 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036067 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036069 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036072 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036075 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036077 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036080 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036082 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036085 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036087 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036089 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036092 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.047969 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036094 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036097 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036099 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036102 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036104 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036107 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036109 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036111 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036114 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036116 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036119 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036121 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036124 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036126 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036128 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036131 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036134 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036136 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036139 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036142 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.048454 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036144 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036147 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036150 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036152 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036155 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036157 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036160 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036162 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036165 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036168 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036170 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036173 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036175 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036177 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036180 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036182 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036184 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036187 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036190 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036192 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.048951 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036194 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.036220 4442 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036356 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036360 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036364 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036366 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036369 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036371 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036374 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036376 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036379 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036382 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036384 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036387 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036389 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036393 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.049428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036395 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036398 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036400 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036403 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036405 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036407 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036410 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036412 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036415 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036417 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036420 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036422 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036425 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036427 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036429 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036431 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036434 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036436 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036439 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.049823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036442 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036446 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036449 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036452 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036454 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036457 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036460 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036463 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036467 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036471 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036473 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036476 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036478 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036481 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036484 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036486 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036488 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036491 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036493 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.050280 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036495 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036498 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036500 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036502 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036505 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036507 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036510 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036512 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036514 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036517 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036519 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036522 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036524 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036527 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036529 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036532 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036534 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036536 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036539 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036541 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.050735 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036544 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036546 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036549 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036551 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036553 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036556 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036558 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036561 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036563 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036566 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036569 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036573 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036576 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036578 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036581 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036583 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036585 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036588 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036590 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036593 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.051289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036595 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036598 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036600 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036602 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036605 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036607 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036610 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036612 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036614 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036617 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036619 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036622 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036624 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036627 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.036648 4442 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036678 4442 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:02:56.051808 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036682 4442 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036684 4442 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036687 4442 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036690 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036723 4442 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036842 4442 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036851 4442 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036863 4442 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036872 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036876 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036881 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036887 4442 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036892 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036896 4442 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036901 4442 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036905 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036912 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036917 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036930 4442 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:02:56.052195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036936 4442 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036941 4442 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036945 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036949 4442 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036955 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036959 4442 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036963 4442 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036967 4442 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036973 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036977 4442 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036981 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036987 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.036998 4442 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037002 4442 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037006 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037013 4442 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037017 4442 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037021 4442 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037029 4442 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037034 4442 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:02:56.052640 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037042 4442 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037047 4442 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037051 4442 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037055 4442 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037064 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037068 4442 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037072 4442 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037077 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037081 4442 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037085 4442 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037093 4442 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037097 4442 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037101 4442 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037105 4442 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037110 4442 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037114 4442 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037123 4442 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037127 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037133 4442 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:02:56.053181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037142 4442 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037147 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037152 4442 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037157 4442 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037162 4442 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037166 4442 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037172 4442 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037179 4442 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037184 4442 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037188 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037197 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037202 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037207 4442 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037213 4442 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037217 4442 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037221 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037226 4442 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037230 4442 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037234 4442 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:02:56.053656 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037238 4442 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037242 4442 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037246 4442 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037256 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037260 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037264 4442 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037268 4442 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037273 4442 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037277 4442 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037282 4442 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037286 4442 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037290 4442 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037294 4442 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037300 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037304 4442 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037309 4442 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037317 4442 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037321 4442 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037326 4442 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037330 4442 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:02:56.054456 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037334 4442 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037341 4442 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037345 4442 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037349 4442 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037353 4442 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037357 4442 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037364 4442 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:02:56.037368 4442 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.037474 4442 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.038962 4442 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.039020 4442 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.039213 4442 server.go:1014] "Starting client certificate rotation" Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.039349 4442 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.039390 4442 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:57:17 +0000 UTC" deadline="2028-07-28 08:22:58.174576092 +0000 UTC" Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.039431 4442 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15293h20m2.135147379s" Oct 30 03:02:56.055311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.039712 4442 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:56.055923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.040838 4442 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:02:56.055923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.042268 4442 log.go:25] "Validated CRI v1 runtime API" Oct 30 03:02:56.055923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.043729 4442 log.go:25] "Validated CRI v1 image API" Oct 30 03:02:56.055923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.044415 4442 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 03:02:56.055923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.045891 4442 fs.go:135] Filesystem UUIDs: map[2d996a21-7135-4f1f-a419-a85ee8c6676e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e022853c-0f66-4f2b-89a4-458f12017129:/dev/nvme0n1p4] Oct 30 03:02:56.056148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.045911 4442 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16671a56f4f848ac148a07362b8ba6c1145934bb57c330abcff45e8f1c78cd6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16671a56f4f848ac148a07362b8ba6c1145934bb57c330abcff45e8f1c78cd6e/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/216d60ca2d895fec6796ef693433e1acd4f050bfdce4b7a6ec31fc5be06c1489/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/216d60ca2d895fec6796ef693433e1acd4f050bfdce4b7a6ec31fc5be06c1489/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30bdbb82a80a556514769a0aa22e1e07cd30f4746d96ed9132351047d4c9f3e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30bdbb82a80a556514769a0aa22e1e07cd30f4746d96ed9132351047d4c9f3e3/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37803cef26cb335d9c1ba97d96e97cdc36bf69aade2685e53418abdbda7ae7b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37803cef26cb335d9c1ba97d96e97cdc36bf69aade2685e53418abdbda7ae7b8/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38532d928dadb938ff3c311632a6402097546df91bbcfac7b98ed6cf9dd12754/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38532d928dadb938ff3c311632a6402097546df91bbcfac7b98ed6cf9dd12754/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e6a7846ae8ffb507824ce42381d0dbeb16426ddea22a22dbdff8af67a9a6149/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e6a7846ae8ffb507824ce42381d0dbeb16426ddea22a22dbdff8af67a9a6149/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a8bb95189927aa9434203255cc30cacd1a98a6dd47b2bb7d1e54e4831856447/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a8bb95189927aa9434203255cc30cacd1a98a6dd47b2bb7d1e54e4831856447/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82bd5ffa790f687c2b2b9d7a7c0939d5a48a02ef705b108160c0462ac27d0932/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82bd5ffa790f687c2b2b9d7a7c0939d5a48a02ef705b108160c0462ac27d0932/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/937c4ea4212df5034f221069c87e913db818f02ccb1bf39ef5618db72ab02377/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/937c4ea4212df5034f221069c87e913db818f02ccb1bf39ef5618db72ab02377/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa0bfb13c8c64fefa115073d60c87b4451f7b8a261dd3a332e37297dd4d42098/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa0bfb13c8c64fefa115073d60c87b4451f7b8a261dd3a332e37297dd4d42098/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c55b339dc474fdf07b3cf868a5755a272fbaba0d871a270328c12bdc620b687f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c55b339dc474fdf07b3cf868a5755a272fbaba0d871a270328c12bdc620b687f/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d52adaf7656f9ec4b858a919f8a57863f076a7780a3e974ade6f2d36cfff4f16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d52adaf7656f9ec4b858a919f8a57863f076a7780a3e974ade6f2d36cfff4f16/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19bd5e4c-0976-4383-82f7-2e8dcb385115/volumes/kubernetes.io~projected/kube-api-access-bnqv9:{mountpoint:/var/lib/kubelet/pods/19bd5e4c-0976-4383-82f7-2e8dcb385115/volumes/kubernetes.io~projected/kube-api-access-bnqv9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19bd5e4c-0976-4383-82f7-2e8dcb385115/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/19bd5e4c-0976-4383-82f7-2e8dcb385115/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20788c87-fbba-4a1c-9d38-f32396e8ed24/volumes/kubernetes.io~projected/kube-api-access-cf6gp:{mountpoint:/var/lib/kubelet/pods/20788c87-fbba-4a1c-9d38-f32396e8ed24/volumes/kubernetes.io~projected/kube-api-access-cf6gp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c3f2dca-e9db-4210-b534-527ed98e7333/volumes/kubernetes.io~projected/kube-api-access-h88zm:{mountpoint:/var/lib/kubelet/pods/2c3f2dca-e9db-4210-b534-527ed98e7333/volumes/kubernetes.io~projected/kube-api-access-h88zm major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fcbb594-f806-4ac7-ab99-ea99ca6c827a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3fcbb594-f806-4ac7-ab99-ea99ca6c827a/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7/volumes/kubernetes.io~projected/kube-api-access-ckj96:{mountpoint:/var/lib/kubelet/pods/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7/volumes/kubernetes.io~projected/kube-api-access-ckj96 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/454e1d2d-f058-4a2b-a420-298761f2ff6a/volumes/kubernetes.io~projected/kube-api-access-57hq8:{mountpoint:/var/lib/kubelet/pods/454e1d2d-f058-4a2b-a420-298761f2ff6a/volumes/kubernetes.io~projected/kube-api-access-57hq8 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d22ace0-adef-488a-a025-83b8270d25d9/volumes/kubernetes.io~projected/kube-api-access-gt8qv:{mountpoint:/var/lib/kubelet/pods/4d22ace0-adef-488a-a025-83b8270d25d9/volumes/kubernetes.io~projected/kube-api-access-gt8qv major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50f7ec66-f78a-405d-95c0-47a26e24a69d/volumes/kubernetes.io~projected/kube-api-access-fmpmr:{mountpoint:/var/lib/kubelet/pods/50f7ec66-f78a-405d-95c0-47a26e24a69d/volumes/kubernetes.io~projected/kube-api-access-fmpmr major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volumes/kubernetes.io~projected/kube-api-access-fszxf:{mountpoint:/var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volumes/kubernetes.io~projected/kube-api-access-fszxf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~projected/kube-api-access-gssmr:{mountpoint:/var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~projected/kube-api-access-gssmr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8434e5c6-4079-46cc-84d5-6672cbac7ad8/volumes/kubernetes.io~projected/kube-api-access-slk6p:{mountpoint:/var/lib/kubelet/pods/8434e5c6-4079-46cc-84d5-6672cbac7ad8/volumes/kubernetes.io~projected/kube-api-access-slk6p major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90b350e9-6998-45b6-882a-17019e89bdf6/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/90b350e9-6998-45b6-882a-17019e89bdf6/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e9460d2-8d0f-4aca-a2a0-860137f1b801/volumes/kubernetes.io~projected/kube-api-access-wwlh9:{mountpoint:/var/lib/kubelet/pods/9e9460d2-8d0f-4aca-a2a0-860137f1b801/volumes/kubernetes.io~projected/kube-api-access-wwlh9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58/volumes/kubernetes.io~projected/kube-api-access-7qn9q:{mountpoint:/var/lib/kubelet/pods/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58/volumes/kubernetes.io~projected/kube-api-access-7qn9q major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc68d015-081c-41bc-b7ec-2443b5693c76/volumes/kubernetes.io~projected/kube-api-access-6bldd:{mountpoint:/var/lib/kubelet/pods/cc68d015-081c-41bc-b7ec-2443b5693c76/volumes/kubernetes.io~projected/kube-api-access-6bldd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dde3589d-5a4c-4176-b493-f5ecfa81349f/volumes/kubernetes.io~projected/kube-api-access-wttlq:{mountpoint:/var/lib/kubelet/pods/dde3589d-5a4c-4176-b493-f5ecfa81349f/volumes/kubernetes.io~projected/kube-api-access-wttlq major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/32e75d030d0890f7aa7cfdd599742044241ebd6728c7697a9dc7877d8a0c14a4/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5049642e8ad4a1c67e5d659a3255210fed02e3f63b64f4fe90e2e5feb08bb120/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3c964f979b5a83f93fe27daa861cca5d6c1e84df8221357fe6f1350cb7568408/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f277b5ddcc5f821f037afe19d23883304c6d4874fe3c59d9dd5977afa2baa6ec/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/abcef2890db9e67e477fdd959892a0b90a9f911055aca0eb3a87645c4dc21842/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6aa52a0ab507364f950b919214fc404304b84a3b50bd507fbd8c02447c9256d0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bbaf721bb5510f8369a3298b981838554d54fa81932bf729afd37d9c35c53939/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ad158fd9cb22002405f73e3cdcce70501ff286bbf0658d8ad8ad99d6d5cd9fe7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5ece2c74d3b343a775a6692a2851ce82ec2d407f24eac1961f0515f58829e9a5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/36077241e963072266be189c1c6ce0a0e99cf32f16acd649ff1a3810979a75be/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/84ecb274d4d0e03c38e00b8b5fba048f31636eb6a415a166d76bc87ce74e728b/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/8e95fb817b60ebbf7d57a0b8066fb1e40453d7a1621475946b2620670943c543/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/01b0be2b84b2019a50c9736152110681eae11011285ad0178b4900a983915fa7/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/04b092fa2148b6e6f86b1614dac504a807f8b914c9750aea8da5f562843274c6/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/63f49372d732ead2ca0403ae9db30c37ba69f2e2dfe2096a29dbeaefa44479bf/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4401483b2d99bcac2ade20a82f38ca18506056af07116ab3338cbfffce9988d6/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c587331327f246907079bc7d605556b673e5ba70fdd8de672b65e473b49baf79/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d03bd821a87ce2d0edb237f91cb147e5485f6a4780bd552d1c371bf52dee6a0b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/33481be36586ee01228d636d294d6d2811fa0e7c47171f0b46d050d40e9fd9ba/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7d5c813d92059847f26716c7aa958a75e21d10ac3a57d1fda59a4b4e1726b779/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/7f5b4b4e9d626203665ecab1a911422d93445167323b598679ddfe617a0609e8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/425e42748630c344f167b77fc8eca6863c08896ab02d84da2d809ef841ed4fc8/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/6629d9f1ac47fe8905b32caa8889e650a7b795d0daf526a1858c6f1182e99426/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/f31372bc7477d5ec92a5bfbbb4c4fded7392fdb8e2c6c9a363882bbad997755b/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/e38c633b0915eefd345ec10fa5cee09d07d81fc1fdfbdb8af46a5469c42021c5/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/79734272cbfa659218ad6737f434ff2cefabc48a930bfa3206e9256c43e5428a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/b45dac17558ec464e58ed02fa683c7f8b0600705c37dce8b7fdc2b6c8d2fc54a/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/b542466618c140d66f0a82bbf43c73b7ddb17c964cae6137bd96f8249a9952d1/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/9e18f58a0186afaf9aa71bff2198b95406a31d8258ad994a16edb7ac88f1a310/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ca4be6da3e4c09da694fce67dbab56ab0b25c8cbe4a6eae2a19f13e55c94df15/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8e912c0663628667e7a222bcf65a11c6338c6bc9913ed54bc12b2feefd9fe35f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fffef742b1520f5b83a32a0361140ad03730ccb87efb53f88713f350d2b735d2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/83d32c9fdc5c1909bedebf2f88fa4422287af54b4ad64c881ae7739b4f005f03/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f621518918bd4c9e68509f05d56200ebf595a2d65891b7f7cf0a465b4e576d5a/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/25dafaecf01fea5b57ad67f9834680139483efc0e549ee4f8d76cccfd7ef965c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/1cae09e75550ebc6a77fc7c1be530baef42ee60ba900302978bbf743503f2eed/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/af091bdc005da73532cb507c6c181b3ee47588724e02270ac1c983912964eb91/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e58de5731686a3ab2baa5c927b2877661121269d0124024253209e957d100931/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a87cce5b8fce9567b280d311fbb5f3b646978dedd1b30479e8b1923d271ceae5/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052203 4442 manager.go:217] Machine: {Timestamp:2026-10-30 03:02:56.051725834 +0000 UTC m=+0.054079022 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3082275 MemoryCapacity:32812003328 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2039908a88285523389463daa3ecc4 SystemUUID:ec203990-8a88-2855-2338-9463daa3ecc4 BootID:0545f8f1-3a29-4e95-8c0d-62f99f24f1f7 Filesystems:[{Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa0bfb13c8c64fefa115073d60c87b4451f7b8a261dd3a332e37297dd4d42098/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d22ace0-adef-488a-a025-83b8270d25d9/volumes/kubernetes.io~projected/kube-api-access-gt8qv DeviceMajor:0 DeviceMinor:225 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/19bd5e4c-0976-4383-82f7-2e8dcb385115/volumes/kubernetes.io~projected/kube-api-access-bnqv9 DeviceMajor:0 DeviceMinor:201 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/90b350e9-6998-45b6-882a-17019e89bdf6/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:220 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82bd5ffa790f687c2b2b9d7a7c0939d5a48a02ef705b108160c0462ac27d0932/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~projected/kube-api-access-gssmr DeviceMajor:0 DeviceMinor:59 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8434e5c6-4079-46cc-84d5-6672cbac7ad8/volumes/kubernetes.io~projected/kube-api-access-slk6p DeviceMajor:0 DeviceMinor:207 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fcbb594-f806-4ac7-ab99-ea99ca6c827a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d52adaf7656f9ec4b858a919f8a57863f076a7780a3e974ade6f2d36cfff4f16/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19bd5e4c-0976-4383-82f7-2e8dcb385115/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/50f7ec66-f78a-405d-95c0-47a26e24a69d/volumes/kubernetes.io~projected/kube-api-access-fmpmr DeviceMajor:0 DeviceMinor:62 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/dde3589d-5a4c-4176-b493-f5ecfa81349f/volumes/kubernetes.io~projected/kube-api-access-wttlq DeviceMajor:0 DeviceMinor:67 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/9e9460d2-8d0f-4aca-a2a0-860137f1b801/volumes/kubernetes.io~projected/kube-api-access-wwlh9 DeviceMajor:0 DeviceMinor:65 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58/volumes/kubernetes.io~projected/kube-api-access-7qn9q DeviceMajor:0 DeviceMinor:197 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7/volumes/kubernetes.io~projected/kube-api-access-ckj96 DeviceMajor:0 DeviceMinor:63 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c55b339dc474fdf07b3cf868a5755a272fbaba0d871a270328c12bdc620b687f/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37803cef26cb335d9c1ba97d96e97cdc36bf69aade2685e53418abdbda7ae7b8/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16405999616 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/655ddd5c-e5dd-4217-b597-6dfb5e33fa73/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/20788c87-fbba-4a1c-9d38-f32396e8ed24/volumes/kubernetes.io~projected/kube-api-access-cf6gp DeviceMajor:0 DeviceMinor:61 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/937c4ea4212df5034f221069c87e913db818f02ccb1bf39ef5618db72ab02377/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16671a56f4f848ac148a07362b8ba6c1145934bb57c330abcff45e8f1c78cd6e/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/585acb34-e959-4d7b-9e5a-7e1e4fbfccea/volumes/kubernetes.io~projected/kube-api-access-fszxf DeviceMajor:0 DeviceMinor:64 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/216d60ca2d895fec6796ef693433e1acd4f050bfdce4b7a6ec31fc5be06c1489/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30bdbb82a80a556514769a0aa22e1e07cd30f4746d96ed9132351047d4c9f3e3/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/454e1d2d-f058-4a2b-a420-298761f2ff6a/volumes/kubernetes.io~projected/kube-api-access-57hq8 DeviceMajor:0 DeviceMinor:60 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/var/lib/kubelet/pods/cc68d015-081c-41bc-b7ec-2443b5693c76/volumes/kubernetes.io~projected/kube-api-access-6bldd DeviceMajor:0 DeviceMinor:66 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a8bb95189927aa9434203255cc30cacd1a98a6dd47b2bb7d1e54e4831856447/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e6a7846ae8ffb507824ce42381d0dbeb16426ddea22a22dbdff8af67a9a6149/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38532d928dadb938ff3c311632a6402097546df91bbcfac7b98ed6cf9dd12754/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005371 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c3f2dca-e9db-4210-b534-527ed98e7333/volumes/kubernetes.io~projected/kube-api-access-h88zm DeviceMajor:0 DeviceMinor:206 Capacity:31633403904 Type:vfs Inodes:4005371 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16671a56f4f848a MacAddress:22:a9:11:89:26:1a Speed:10000 Mtu:8901} {Name:30bdbb82a80a556 MacAddress:ea:12:a4:8f:de:86 Speed:10000 Mtu:8901} {Name:82bd5ffa790f687 MacAddress:da:d7:58:86:0f:2b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:67:cb:bd Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:1c:3d:d8:bf:2a Speed:0 Mtu:8901} {Name:c55b339dc474fdf MacAddress:56:43:3f:0a:7f:c4 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:67:cb:bd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:a8:03:93:03:67 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:75:02:1a:19:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812003328 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052437 4442 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052478 4442 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052788 4442 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052810 4442 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052946 4442 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052954 4442 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052967 4442 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.052984 4442 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053058 4442 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053218 4442 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053312 4442 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053345 4442 kubelet.go:493] "Attempting to sync node with API server" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053359 4442 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053370 4442 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053379 4442 kubelet.go:405] "Adding apiserver pod source" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.053387 4442 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.055721 4442 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 03:02:56.056352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.055800 4442 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:56.057251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.057015 4442 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 03:02:56.059105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059086 4442 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 03:02:56.059105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059104 4442 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:02:56.059258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059231 4442 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 03:02:56.059464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059449 4442 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 03:02:56.059535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059489 4442 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 03:02:56.059614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059603 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 03:02:56.059667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059623 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 03:02:56.059667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059634 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 03:02:56.059667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059644 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 03:02:56.059667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059652 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 03:02:56.059667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059662 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059671 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059680 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059691 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059701 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059713 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059728 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059777 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 03:02:56.059918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.059788 4442 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 03:02:56.063190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.063173 4442 server.go:1263] "Started kubelet" Oct 30 03:02:56.063284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.063259 4442 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 03:02:56.063349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.063265 4442 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 03:02:56.063349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.063340 4442 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 03:02:56.063652 ip-10-0-134-92 systemd[1]: Started Kubernetes Kubelet. Oct 30 03:02:56.063839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.063819 4442 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 03:02:56.064413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.064397 4442 server.go:317] "Adding debug handlers to kubelet server" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.066480 4442 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.066513 4442 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.066609 4442 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.066598 4442 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:57:18 +0000 UTC" deadline="2028-05-25 13:33:13.784638438 +0000 UTC" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.066626 4442 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.066634 4442 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13762h30m17.718007807s" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:56.066691 4442 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-92.ec2.internal\" not found" Oct 30 03:02:56.067985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.066731 4442 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068191 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="655ddd5c-e5dd-4217-b597-6dfb5e33fa73" volumeName="kubernetes.io/projected/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-kube-api-access-gssmr" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068228 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e9460d2-8d0f-4aca-a2a0-860137f1b801" volumeName="kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-binary-copy" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068237 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc68d015-081c-41bc-b7ec-2443b5693c76" volumeName="kubernetes.io/projected/cc68d015-081c-41bc-b7ec-2443b5693c76-kube-api-access-6bldd" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068245 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19bd5e4c-0976-4383-82f7-2e8dcb385115" volumeName="kubernetes.io/secret/19bd5e4c-0976-4383-82f7-2e8dcb385115-cert" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068253 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585acb34-e959-4d7b-9e5a-7e1e4fbfccea" volumeName="kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-config" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068261 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585acb34-e959-4d7b-9e5a-7e1e4fbfccea" volumeName="kubernetes.io/secret/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068269 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58" volumeName="kubernetes.io/configmap/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-config-volume" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068279 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c3f2dca-e9db-4210-b534-527ed98e7333" volumeName="kubernetes.io/projected/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-api-access-h88zm" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068290 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fcbb594-f806-4ac7-ab99-ea99ca6c827a" volumeName="kubernetes.io/configmap/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-konnectivity-ca" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068300 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585acb34-e959-4d7b-9e5a-7e1e4fbfccea" volumeName="kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-script-lib" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068310 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dde3589d-5a4c-4176-b493-f5ecfa81349f" volumeName="kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-cni-binary-copy" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068318 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e9460d2-8d0f-4aca-a2a0-860137f1b801" volumeName="kubernetes.io/projected/9e9460d2-8d0f-4aca-a2a0-860137f1b801-kube-api-access-wwlh9" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068325 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454e1d2d-f058-4a2b-a420-298761f2ff6a" volumeName="kubernetes.io/projected/454e1d2d-f058-4a2b-a420-298761f2ff6a-kube-api-access-57hq8" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068339 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e9460d2-8d0f-4aca-a2a0-860137f1b801" volumeName="kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068348 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58" volumeName="kubernetes.io/empty-dir/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-tmp-dir" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068355 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20788c87-fbba-4a1c-9d38-f32396e8ed24" volumeName="kubernetes.io/configmap/20788c87-fbba-4a1c-9d38-f32396e8ed24-iptables-alerter-script" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068362 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fcbb594-f806-4ac7-ab99-ea99ca6c827a" volumeName="kubernetes.io/secret/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-agent-certs" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068371 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dde3589d-5a4c-4176-b493-f5ecfa81349f" volumeName="kubernetes.io/projected/dde3589d-5a4c-4176-b493-f5ecfa81349f-kube-api-access-wttlq" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068377 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c3f2dca-e9db-4210-b534-527ed98e7333" volumeName="kubernetes.io/empty-dir/2c3f2dca-e9db-4210-b534-527ed98e7333-data-volume" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068385 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90b350e9-6998-45b6-882a-17019e89bdf6" volumeName="kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068392 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e9460d2-8d0f-4aca-a2a0-860137f1b801" volumeName="kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068398 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20788c87-fbba-4a1c-9d38-f32396e8ed24" volumeName="kubernetes.io/projected/20788c87-fbba-4a1c-9d38-f32396e8ed24-kube-api-access-cf6gp" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068406 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7" volumeName="kubernetes.io/empty-dir/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-tmp-dir" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068413 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7" volumeName="kubernetes.io/projected/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-kube-api-access-ckj96" seLinuxMountContext="" Oct 30 03:02:56.068406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068421 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d22ace0-adef-488a-a025-83b8270d25d9" volumeName="kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068429 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f7ec66-f78a-405d-95c0-47a26e24a69d" volumeName="kubernetes.io/projected/50f7ec66-f78a-405d-95c0-47a26e24a69d-kube-api-access-fmpmr" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068436 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="655ddd5c-e5dd-4217-b597-6dfb5e33fa73" volumeName="kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-tmp" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068443 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19bd5e4c-0976-4383-82f7-2e8dcb385115" volumeName="kubernetes.io/projected/19bd5e4c-0976-4383-82f7-2e8dcb385115-kube-api-access-bnqv9" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068452 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50f7ec66-f78a-405d-95c0-47a26e24a69d" volumeName="kubernetes.io/configmap/50f7ec66-f78a-405d-95c0-47a26e24a69d-serviceca" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068468 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585acb34-e959-4d7b-9e5a-7e1e4fbfccea" volumeName="kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-env-overrides" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068474 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="585acb34-e959-4d7b-9e5a-7e1e4fbfccea" volumeName="kubernetes.io/projected/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-kube-api-access-fszxf" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068481 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8434e5c6-4079-46cc-84d5-6672cbac7ad8" volumeName="kubernetes.io/projected/8434e5c6-4079-46cc-84d5-6672cbac7ad8-kube-api-access-slk6p" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068490 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58" volumeName="kubernetes.io/projected/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-kube-api-access-7qn9q" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068497 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dde3589d-5a4c-4176-b493-f5ecfa81349f" volumeName="kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-daemon-config" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068505 4442 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="655ddd5c-e5dd-4217-b597-6dfb5e33fa73" volumeName="kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-tuned" seLinuxMountContext="" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068512 4442 reconstruct.go:97] "Volume reconstruction finished" Oct 30 03:02:56.069589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.068517 4442 reconciler.go:29] "Reconciler: start to sync state" Oct 30 03:02:56.070328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.069852 4442 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 03:02:56.070328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.069868 4442 factory.go:55] Registering systemd factory Oct 30 03:02:56.070328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.069878 4442 factory.go:223] Registration of the systemd container factory successfully Oct 30 03:02:56.071175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.071158 4442 factory.go:153] Registering CRI-O factory Oct 30 03:02:56.071266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.071181 4442 factory.go:223] Registration of the crio container factory successfully Oct 30 03:02:56.071266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.071212 4442 factory.go:103] Registering Raw factory Oct 30 03:02:56.071266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.071226 4442 manager.go:1199] Started watching for new ooms in manager Oct 30 03:02:56.071741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.071723 4442 manager.go:319] Starting recovery of all containers Oct 30 03:02:56.081992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.081973 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:02:56.082122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.081969 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:02:56.097222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.097182 4442 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 03:02:56.099244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.099217 4442 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 03:02:56.099244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.099246 4442 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 03:02:56.099393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.099272 4442 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 03:02:56.099393 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:56.099319 4442 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 03:02:56.111671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.111653 4442 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:56.111919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.111902 4442 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 03:02:56.112910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.112890 4442 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:56.115498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.115474 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="cb4add2f0709a4d315d6dc7f39089676ed4d3606ea83ccc82a0e27ab7cc46309" exitCode=0 Oct 30 03:02:56.115498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.115498 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="0fcd1e8fc72ad168c1860893775cf1c064c7e6b0942b3ce0ee3f675ee416ef9a" exitCode=0 Oct 30 03:02:56.115600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.115506 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="722d736b88beaf4b63cf75b788a0b21235a4a5baee97c098dd87bb529efd52c9" exitCode=0 Oct 30 03:02:56.115600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.115511 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="5c2399eaec814d0bcf816faff7326e1a3363c3bdd5e7bf542647b805f147ce59" exitCode=0 Oct 30 03:02:56.115600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.115516 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="87583a3a207dded8942f5777c333b19bde6aa42be006611f9a3361d6948d3183" exitCode=0 Oct 30 03:02:56.115600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.115520 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e9460d2-8d0f-4aca-a2a0-860137f1b801" containerID="e5423947d18ded5f1fcaac313b94a6ce61ad30b8758895ce1114692e1b817f79" exitCode=0 Oct 30 03:02:56.119394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.119366 4442 generic.go:356] "Generic (PLEG): container finished" podID="dca45c9b1760cdbab09f6b1b711a4a17" containerID="832ec5fda840c2ec4df9faa6923699414cf8bfe16ccfbff7ed32f8d96aa29f03" exitCode=0 Oct 30 03:02:56.124170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.124143 4442 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:56.124306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.124284 4442 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:02:56.127569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.127552 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:02:56.127950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.127930 4442 generic.go:356] "Generic (PLEG): container finished" podID="585acb34-e959-4d7b-9e5a-7e1e4fbfccea" containerID="5002bbb2b237ed1479172b4defee2ff013d8305ffcdd720b8b978e3d848a6c2a" exitCode=1 Oct 30 03:02:56.137604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.137588 4442 manager.go:324] Recovery completed Oct 30 03:02:56.152424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152399 4442 cpu_manager.go:226] "Starting" policy="none" Oct 30 03:02:56.152424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152413 4442 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 03:02:56.152424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152427 4442 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 03:02:56.152573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152544 4442 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 03:02:56.152977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152554 4442 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 03:02:56.153025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152980 4442 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 03:02:56.153025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152986 4442 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 03:02:56.153025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.152992 4442 policy_none.go:50] "Start" Oct 30 03:02:56.153025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.153000 4442 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 03:02:56.153025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.153012 4442 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 03:02:56.153150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.153137 4442 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 03:02:56.153150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.153144 4442 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 03:02:56.153150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.153149 4442 policy_none.go:44] "Start" Oct 30 03:02:56.157015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.156990 4442 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 03:02:56.160262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160248 4442 manager.go:341] "Starting Device Plugin manager" Oct 30 03:02:56.160369 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:56.160276 4442 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 03:02:56.160369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160287 4442 server.go:87] "Starting device plugin registration server" Oct 30 03:02:56.160603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160589 4442 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 03:02:56.160676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160603 4442 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 03:02:56.160731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160700 4442 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 03:02:56.160848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160815 4442 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 03:02:56.160902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160858 4442 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 03:02:56.160902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160865 4442 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 03:02:56.161009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.160940 4442 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:02:56.160848028Z","UUID":"33cfcc41-66f6-4040-9332-8a8bc47fabe6","Handler":null,"Name":"","Endpoint":""} Oct 30 03:02:56.161940 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:56.161912 4442 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 03:02:56.162303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.162271 4442 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 03:02:56.162303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.162303 4442 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 03:02:56.199551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.199506 4442 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal"] Oct 30 03:02:56.200051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.200027 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.200126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.200065 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.200126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.200076 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" event={"ID":"7c18c0405adfb2995cb6b447bf5eb736","Type":"ContainerStarted","Data":"1982beab2c1b65291da7031eb906596a05bbd7a3275c2f95853b0ec21f8b369e"} Oct 30 03:02:56.200126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.200119 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" event={"ID":"7c18c0405adfb2995cb6b447bf5eb736","Type":"ContainerStarted","Data":"aba6b1d234747cece1cd83580c7519cbb5b0286a252713185c3caa533d6ffd8d"} Oct 30 03:02:56.200234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.200164 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" event={"ID":"dca45c9b1760cdbab09f6b1b711a4a17","Type":"ContainerStarted","Data":"674a113008f39ca43dd97d1b0d3e4e34d57b681f420a8366b5dcc0bbfa6256ff"} Oct 30 03:02:56.200234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.200178 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" event={"ID":"dca45c9b1760cdbab09f6b1b711a4a17","Type":"ContainerDied","Data":"832ec5fda840c2ec4df9faa6923699414cf8bfe16ccfbff7ed32f8d96aa29f03"} Oct 30 03:02:56.200234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.200190 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" event={"ID":"dca45c9b1760cdbab09f6b1b711a4a17","Type":"ContainerStarted","Data":"cfe6103b31860b73a63c039669dc4ff5da47447cebfa7d577541ab9b3aaaefdb"} Oct 30 03:02:56.221177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.221162 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:56.221261 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:56.221201 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.221694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.221680 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:56.221728 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:56.221716 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.261480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.261462 4442 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:02:56.262717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.262703 4442 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:02:56.262812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.262729 4442 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:02:56.262812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.262739 4442 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-92.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:02:56.262812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.262783 4442 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.294282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.294238 4442 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.294342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.294291 4442 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.312514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.312487 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c18c0405adfb2995cb6b447bf5eb736-config\") pod \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" (UID: \"7c18c0405adfb2995cb6b447bf5eb736\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.312610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.312520 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.312610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.312545 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.413549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.413517 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c18c0405adfb2995cb6b447bf5eb736-config\") pod \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" (UID: \"7c18c0405adfb2995cb6b447bf5eb736\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.413549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.413554 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.413720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.413578 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.413720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.413632 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.413720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.413661 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dca45c9b1760cdbab09f6b1b711a4a17-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" (UID: \"dca45c9b1760cdbab09f6b1b711a4a17\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:56.413720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:56.413631 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7c18c0405adfb2995cb6b447bf5eb736-config\") pod \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" (UID: \"7c18c0405adfb2995cb6b447bf5eb736\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:57.053782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.053741 4442 apiserver.go:52] "Watching apiserver" Oct 30 03:02:57.124481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.124457 4442 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 03:02:57.124918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.124899 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-9fhjj","openshift-multus/multus-tpl8m","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7","openshift-cluster-node-tuning-operator/tuned-gnxzn","openshift-multus/network-metrics-daemon-qxx87","openshift-network-operator/iptables-alerter-hq9jd","openshift-ingress-canary/ingress-canary-pmfcp","openshift-insights/insights-runtime-extractor-cp2d8","kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal","openshift-dns/dns-default-9625v","openshift-ovn-kubernetes/ovnkube-node-qtbgx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal","openshift-network-diagnostics/network-check-target-6szch","openshift-dns/node-resolver-jj8ks","openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx","kube-system/global-pull-secret-syncer-m266j","kube-system/konnectivity-agent-j2q75","openshift-image-registry/node-ca-shbmk"] Oct 30 03:02:57.126603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.126580 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:57.131645 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.131621 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.132102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.132087 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.137033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.137015 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:57.137187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.137174 4442 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:57.140408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.140391 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.141160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141144 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.141232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141169 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 03:02:57.141232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141149 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.141232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141184 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.141232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141197 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.141232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141206 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.141232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141220 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141388 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141419 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141429 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141439 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141391 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-shnp4\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141471 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f9hw2\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141443 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 03:02:57.141483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141453 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.141776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141513 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wnh6p\"" Oct 30 03:02:57.141776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141537 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 03:02:57.141776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141591 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6fpwt\"" Oct 30 03:02:57.141776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141624 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.141776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.141685 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 03:02:57.150254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.150241 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mpbq7\"" Oct 30 03:02:57.152214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152200 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nc7qz\"" Oct 30 03:02:57.152214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152204 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tc5sd\"" Oct 30 03:02:57.152553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152536 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 03:02:57.152553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152549 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.152662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152571 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.152662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152583 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.152739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152625 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 03:02:57.152908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152890 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bl9p4\"" Oct 30 03:02:57.152970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152936 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.152970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152944 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 03:02:57.152970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152947 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 03:02:57.152970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152955 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 03:02:57.152970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152947 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g7q2t\"" Oct 30 03:02:57.152970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152971 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 03:02:57.153185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.152996 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 03:02:57.153185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153007 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.153185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153110 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.153351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153330 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 03:02:57.153351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153344 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 03:02:57.153507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153375 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j94bv\"" Oct 30 03:02:57.153507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153444 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.153507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153498 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.153507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153501 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153521 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-hfhbp\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153555 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153570 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qj6bv\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153582 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-69wt9\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153587 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153591 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153590 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153601 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dgj97\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153652 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153657 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vnb2g\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153656 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hhqhj\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153674 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 03:02:57.153879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.153768 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 03:02:57.156867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.156853 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:57.156935 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.156889 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-92.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-92.ec2.internal" Oct 30 03:02:57.157682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.157671 4442 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:02:57.157734 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.157697 4442 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-92.ec2.internal" Oct 30 03:02:57.168404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.168388 4442 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 03:02:57.218808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.218789 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-systemd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.218890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.218815 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-bin\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.218890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.218833 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.218890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.218851 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmpmr\" (UniqueName: \"kubernetes.io/projected/50f7ec66-f78a-405d-95c0-47a26e24a69d-kube-api-access-fmpmr\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:57.218890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.218867 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-slash\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.218890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.218883 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.219105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219021 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-daemon-config\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.219151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219102 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-multus-certs\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.219223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219204 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.219223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219219 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.219306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219245 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-binary-copy\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.219306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219250 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-device-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.219306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219270 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-daemon-config\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.219306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219287 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-kubernetes\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219317 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-systemd\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219342 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-k8s-cni-cncf-io\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.219410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219366 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wttlq\" (UniqueName: \"kubernetes.io/projected/dde3589d-5a4c-4176-b493-f5ecfa81349f-kube-api-access-wttlq\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.219410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219396 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bd5e4c-0976-4383-82f7-2e8dcb385115-cert\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:57.219513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219456 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-sys\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219472 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-lib-modules\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219488 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-tuned\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219503 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-agent-certs\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219518 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qn9q\" (UniqueName: \"kubernetes.io/projected/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-kube-api-access-7qn9q\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219545 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovn-node-metrics-cert\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219571 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219601 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-host\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219618 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c3f2dca-e9db-4210-b534-527ed98e7333-crio-socket\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219636 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-kubelet\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219641 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19bd5e4c-0976-4383-82f7-2e8dcb385115-cert\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:57.219667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219657 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-netns\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.219989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219732 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnqv9\" (UniqueName: \"kubernetes.io/projected/19bd5e4c-0976-4383-82f7-2e8dcb385115-kube-api-access-bnqv9\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:57.219989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219867 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-run\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219883 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-tuned\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.219989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219873 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-agent-certs\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:57.219989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219901 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f2dca-e9db-4210-b534-527ed98e7333-data-volume\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.219989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219926 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovn-node-metrics-cert\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.219989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.219988 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2c3f2dca-e9db-4210-b534-527ed98e7333-data-volume\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.220212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220041 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20788c87-fbba-4a1c-9d38-f32396e8ed24-host-slash\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:57.220212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220072 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-konnectivity-ca\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:57.220212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220088 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-system-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.220212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220103 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-dbus\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.220212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220118 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gssmr\" (UniqueName: \"kubernetes.io/projected/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-kube-api-access-gssmr\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.220212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220148 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bldd\" (UniqueName: \"kubernetes.io/projected/cc68d015-081c-41bc-b7ec-2443b5693c76-kube-api-access-6bldd\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:57.220449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220327 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f7ec66-f78a-405d-95c0-47a26e24a69d-host\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:57.220449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220378 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3fcbb594-f806-4ac7-ab99-ea99ca6c827a-konnectivity-ca\") pod \"konnectivity-agent-j2q75\" (UID: \"3fcbb594-f806-4ac7-ab99-ea99ca6c827a\") " pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:02:57.220449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220432 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.220541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220486 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-tmp-dir\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:57.220571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220527 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckj96\" (UniqueName: \"kubernetes.io/projected/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-kube-api-access-ckj96\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:57.220606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220566 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cnibin\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.220606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220572 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-tmp-dir\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:57.220606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220591 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-socket-dir-parent\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.220687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220616 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-etc-kubernetes\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.220687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220638 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-sys-fs\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.220687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220660 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-modprobe-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.220795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220707 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-tmp-dir\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.220795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220737 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-log-socket\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.220795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220777 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.220870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220793 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-tmp-dir\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.220870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220803 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.220870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220862 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlh9\" (UniqueName: \"kubernetes.io/projected/9e9460d2-8d0f-4aca-a2a0-860137f1b801-kube-api-access-wwlh9\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.220980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220898 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-cni-binary-copy\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.220980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220924 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-netns\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.220980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220948 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-netd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.220996 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-bin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.221125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221014 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-kubelet-config\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.221125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221033 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.221125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221065 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:57.221125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221100 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-config\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221152 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-script-lib\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221239 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dde3589d-5a4c-4176-b493-f5ecfa81349f-cni-binary-copy\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.221350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221266 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-registration-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.221350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221301 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50f7ec66-f78a-405d-95c0-47a26e24a69d-serviceca\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:57.221350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221334 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-node-log\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221348 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-hosts-file\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-cnibin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221396 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-hostroot\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221412 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-script-lib\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221423 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221450 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-ovnkube-config\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221459 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88zm\" (UniqueName: \"kubernetes.io/projected/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-api-access-h88zm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221494 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/50f7ec66-f78a-405d-95c0-47a26e24a69d-serviceca\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221518 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20788c87-fbba-4a1c-9d38-f32396e8ed24-iptables-alerter-script\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221538 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-etc-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221559 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-socket-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.221866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221588 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-var-lib-kubelet\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.221866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221667 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf6gp\" (UniqueName: \"kubernetes.io/projected/20788c87-fbba-4a1c-9d38-f32396e8ed24-kube-api-access-cf6gp\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:57.221866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221696 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-var-lib-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221747 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.221866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221774 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20788c87-fbba-4a1c-9d38-f32396e8ed24-iptables-alerter-script\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:57.221866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221854 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-env-overrides\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.222050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221883 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszxf\" (UniqueName: \"kubernetes.io/projected/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-kube-api-access-fszxf\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.222050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.221960 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-system-cni-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.222050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222017 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-multus\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.222050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222034 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-env-overrides\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.222162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222077 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-kubelet\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.222162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222096 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hq8\" (UniqueName: \"kubernetes.io/projected/454e1d2d-f058-4a2b-a420-298761f2ff6a-kube-api-access-57hq8\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.222162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222111 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-ovn\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.222162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222124 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-systemd-units\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.222267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222209 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.222295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222267 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slk6p\" (UniqueName: \"kubernetes.io/projected/8434e5c6-4079-46cc-84d5-6672cbac7ad8-kube-api-access-slk6p\") pod \"migrator-66df5d496b-fn9bx\" (UID: \"8434e5c6-4079-46cc-84d5-6672cbac7ad8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" Oct 30 03:02:57.222323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222305 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.222414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222399 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.222499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222434 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-config-volume\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.222566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222535 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.222612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222564 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/90b350e9-6998-45b6-882a-17019e89bdf6-original-pull-secret\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.222612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222575 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.222737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222640 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-config-volume\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.222737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222694 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-conf-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.222737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222711 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.222737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222722 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysconfig\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.222737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222737 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-conf\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.222995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222771 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-tmp\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.222995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222795 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:57.222995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.222826 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-tmp\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.235907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.235853 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wttlq\" (UniqueName: \"kubernetes.io/projected/dde3589d-5a4c-4176-b493-f5ecfa81349f-kube-api-access-wttlq\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.235907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.235889 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmpmr\" (UniqueName: \"kubernetes.io/projected/50f7ec66-f78a-405d-95c0-47a26e24a69d-kube-api-access-fmpmr\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:57.236338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.236310 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnqv9\" (UniqueName: \"kubernetes.io/projected/19bd5e4c-0976-4383-82f7-2e8dcb385115-kube-api-access-bnqv9\") pod \"ingress-canary-pmfcp\" (UID: \"19bd5e4c-0976-4383-82f7-2e8dcb385115\") " pod="openshift-ingress-canary/ingress-canary-pmfcp" Oct 30 03:02:57.236490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.236472 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bldd\" (UniqueName: \"kubernetes.io/projected/cc68d015-081c-41bc-b7ec-2443b5693c76-kube-api-access-6bldd\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:57.236606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.236571 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qn9q\" (UniqueName: \"kubernetes.io/projected/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-kube-api-access-7qn9q\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.237029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.237007 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlh9\" (UniqueName: \"kubernetes.io/projected/9e9460d2-8d0f-4aca-a2a0-860137f1b801-kube-api-access-wwlh9\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.237419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.237400 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckj96\" (UniqueName: \"kubernetes.io/projected/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-kube-api-access-ckj96\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:57.237518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.237440 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gssmr\" (UniqueName: \"kubernetes.io/projected/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-kube-api-access-gssmr\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.239131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.239111 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszxf\" (UniqueName: \"kubernetes.io/projected/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-kube-api-access-fszxf\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.239251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.239235 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slk6p\" (UniqueName: \"kubernetes.io/projected/8434e5c6-4079-46cc-84d5-6672cbac7ad8-kube-api-access-slk6p\") pod \"migrator-66df5d496b-fn9bx\" (UID: \"8434e5c6-4079-46cc-84d5-6672cbac7ad8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fn9bx" Oct 30 03:02:57.239320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.239250 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf6gp\" (UniqueName: \"kubernetes.io/projected/20788c87-fbba-4a1c-9d38-f32396e8ed24-kube-api-access-cf6gp\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:57.239452 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.239436 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt8qv\" (UniqueName: \"kubernetes.io/projected/4d22ace0-adef-488a-a025-83b8270d25d9-kube-api-access-gt8qv\") pod \"network-check-target-6szch\" (UID: \"4d22ace0-adef-488a-a025-83b8270d25d9\") " pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:57.239982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.239962 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88zm\" (UniqueName: \"kubernetes.io/projected/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-api-access-h88zm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.240021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.240009 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hq8\" (UniqueName: \"kubernetes.io/projected/454e1d2d-f058-4a2b-a420-298761f2ff6a-kube-api-access-57hq8\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.323399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323337 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysconfig\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.323399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323365 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-conf\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.323399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323381 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:57.323399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323395 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-systemd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.323596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323463 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysconfig\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.323596 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.323514 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:57.323596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323524 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-systemd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.323596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323570 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-bin\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.323596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-slash\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323611 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-multus-certs\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323627 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323658 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-slash\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.323663 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:57.823629887 +0000 UTC m=+1.825983079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : secret "metrics-daemon-secret" not found Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323676 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-conf\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323684 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323694 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-bin\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323704 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-device-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323731 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-multus-certs\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323780 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-device-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323787 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-kubernetes\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.323819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323816 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-kubernetes\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323838 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-systemd\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323859 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-k8s-cni-cncf-io\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323876 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-sys\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323890 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-lib-modules\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323915 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323931 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-k8s-cni-cncf-io\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323935 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-sys\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323936 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-host\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323977 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-systemd\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323988 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323984 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-host\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.323995 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c3f2dca-e9db-4210-b534-527ed98e7333-crio-socket\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324025 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-kubelet\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324044 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2c3f2dca-e9db-4210-b534-527ed98e7333-crio-socket\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324051 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-netns\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324028 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-lib-modules\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324076 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-run\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324118 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-kubelet\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324121 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-netns\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324124 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20788c87-fbba-4a1c-9d38-f32396e8ed24-host-slash\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324153 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-run\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324152 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20788c87-fbba-4a1c-9d38-f32396e8ed24-host-slash\") pod \"iptables-alerter-hq9jd\" (UID: \"20788c87-fbba-4a1c-9d38-f32396e8ed24\") " pod="openshift-network-operator/iptables-alerter-hq9jd" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324168 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-system-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324184 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-dbus\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324209 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f7ec66-f78a-405d-95c0-47a26e24a69d-host\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324215 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-dbus\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.324237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324231 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324253 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cnibin\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324265 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-system-cni-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324292 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-cnibin\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324270 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-socket-dir-parent\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324252 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50f7ec66-f78a-405d-95c0-47a26e24a69d-host\") pod \"node-ca-shbmk\" (UID: \"50f7ec66-f78a-405d-95c0-47a26e24a69d\") " pod="openshift-image-registry/node-ca-shbmk" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324311 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-socket-dir-parent\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.324332 4442 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324327 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-etc-kubernetes\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324344 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-etc-kubernetes\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324361 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-sys-fs\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.324387 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:57.824371747 +0000 UTC m=+1.826724939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324403 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-modprobe-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324412 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-sys-fs\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324428 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-log-socket\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324447 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324463 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324480 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-netns\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324499 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-netd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324513 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324521 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-bin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324499 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-log-socket\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324544 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-run-netns\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324551 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-cni-netd\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324523 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-modprobe-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324548 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-kubelet-config\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324550 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-bin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324599 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324579 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/90b350e9-6998-45b6-882a-17019e89bdf6-kubelet-config\") pod \"global-pull-secret-syncer-m266j\" (UID: \"90b350e9-6998-45b6-882a-17019e89bdf6\") " pod="kube-system/global-pull-secret-syncer-m266j" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324623 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-registration-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.324633 4442 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324654 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-registration-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324654 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-node-log\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.324672 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:57.82466425 +0000 UTC m=+1.827017490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324691 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-node-log\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324689 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-hosts-file\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324726 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-cnibin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324740 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-hostroot\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324770 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324783 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7-hosts-file\") pod \"node-resolver-jj8ks\" (UID: \"436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7\") " pod="openshift-dns/node-resolver-jj8ks" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324785 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-etc-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324793 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-cnibin\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324809 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-etc-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324811 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-hostroot\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324823 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-socket-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324840 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-var-lib-kubelet\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.324849 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324854 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-var-lib-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324880 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.324901 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:57.824887436 +0000 UTC m=+1.827240616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324915 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324919 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-var-lib-openvswitch\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324929 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-socket-dir\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324933 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-system-cni-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324943 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-var-lib-kubelet\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324952 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-system-cni-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324957 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-multus\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324971 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-kubelet\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324988 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-ovn\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.324999 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-cni-multus\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325014 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-systemd-units\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325031 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325044 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-run-ovn\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325030 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-host-var-lib-kubelet\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325049 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325069 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/585acb34-e959-4d7b-9e5a-7e1e4fbfccea-systemd-units\") pod \"ovnkube-node-qtbgx\" (UID: \"585acb34-e959-4d7b-9e5a-7e1e4fbfccea\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325080 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/454e1d2d-f058-4a2b-a420-298761f2ff6a-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgbz7\" (UID: \"454e1d2d-f058-4a2b-a420-298761f2ff6a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgbz7" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325091 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325119 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-conf-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325159 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/655ddd5c-e5dd-4217-b597-6dfb5e33fa73-etc-sysctl-d\") pod \"tuned-gnxzn\" (UID: \"655ddd5c-e5dd-4217-b597-6dfb5e33fa73\") " pod="openshift-cluster-node-tuning-operator/tuned-gnxzn" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325168 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e9460d2-8d0f-4aca-a2a0-860137f1b801-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9fhjj\" (UID: \"9e9460d2-8d0f-4aca-a2a0-860137f1b801\") " pod="openshift-multus/multus-additional-cni-plugins-9fhjj" Oct 30 03:02:57.325350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.325172 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dde3589d-5a4c-4176-b493-f5ecfa81349f-multus-conf-dir\") pod \"multus-tpl8m\" (UID: \"dde3589d-5a4c-4176-b493-f5ecfa81349f\") " pod="openshift-multus/multus-tpl8m" Oct 30 03:02:57.701418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.701329 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:57.829097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.829066 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.829271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.829103 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:57.829271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.829131 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:57.829271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:57.829157 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:57.829271 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829206 4442 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:57.829271 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829236 4442 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:57.829271 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829236 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:57.829271 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829248 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:57.829499 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829279 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:58.829264186 +0000 UTC m=+2.831617395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:57.829499 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829297 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:58.829285615 +0000 UTC m=+2.831638800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:57.829499 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829312 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:58.829306214 +0000 UTC m=+2.831659389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:57.829499 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:57.829325 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:02:58.829317034 +0000 UTC m=+2.831670209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : secret "metrics-daemon-secret" not found Oct 30 03:02:58.304404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.304371 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:58.506515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.506472 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:58.508428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.508408 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:02:58.836078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.836042 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:02:58.836259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.836083 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:58.836259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.836106 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:02:58.836259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.836133 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:02:58.836259 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836185 4442 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 03:02:58.836259 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836237 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:02:58.836259 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836244 4442 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:02:58.836464 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836269 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:02:58.836464 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836270 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.836248582 +0000 UTC m=+4.838601773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : configmap "kube-rbac-proxy" not found Oct 30 03:02:58.836464 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836319 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.836307091 +0000 UTC m=+4.838660280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:02:58.836464 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836337 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.836328577 +0000 UTC m=+4.838681760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : secret "metrics-daemon-secret" not found Oct 30 03:02:58.836464 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:02:58.836351 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:00.836343224 +0000 UTC m=+4.838696399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:02:58.904922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:58.904893 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:59.116374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:59.116289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:59.701419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:59.701378 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:02:59.738739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:02:59.738716 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:03:00.329878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.329845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:00.514765 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.514724 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:00.846699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.846667 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:00.851688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.851670 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:03:00.851762 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.851706 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:03:00.851762 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.851734 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:03:00.851830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.851785 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:03:00.851863 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:00.851844 4442 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 03:03:00.851863 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:00.851844 4442 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 03:03:00.851923 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:00.851899 4442 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 03:03:00.851923 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:00.851909 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs podName:cc68d015-081c-41bc-b7ec-2443b5693c76 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:04.851893711 +0000 UTC m=+8.854246900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs") pod "network-metrics-daemon-qxx87" (UID: "cc68d015-081c-41bc-b7ec-2443b5693c76") : secret "metrics-daemon-secret" not found Oct 30 03:03:00.851998 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:00.851926 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls podName:2c3f2dca-e9db-4210-b534-527ed98e7333 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:04.851917813 +0000 UTC m=+8.854270988 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cp2d8" (UID: "2c3f2dca-e9db-4210-b534-527ed98e7333") : secret "insights-runtime-extractor-tls" not found Oct 30 03:03:00.851998 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:00.851948 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls podName:c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58 nodeName:}" failed. No retries permitted until 2026-10-30 03:03:04.851935416 +0000 UTC m=+8.854288591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls") pod "dns-default-9625v" (UID: "c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58") : secret "dns-default-metrics-tls" not found Oct 30 03:03:00.852328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.852311 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2c3f2dca-e9db-4210-b534-527ed98e7333-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:03:00.946088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:00.946054 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:01.110781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:01.110692 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:01.223811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:01.223781 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:01.707674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:01.707642 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:02.308711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:02.308678 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:02.507747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:02.507715 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:02.775875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:02.775849 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:03:02.776086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:02.776071 4442 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:03:02.907079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:02.907051 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:03.010608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:03.010582 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:03.107225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:03.107158 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:03.207091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:03.207063 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:03.335828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:03.335798 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:03.510547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:03.510521 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:04.302107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.302074 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:04.708214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.708131 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:04.879632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.879601 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:03:04.879840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.879640 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:03:04.879840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.879786 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:03:04.880019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.880000 4442 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 03:03:04.882264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.882237 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2c3f2dca-e9db-4210-b534-527ed98e7333-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cp2d8\" (UID: \"2c3f2dca-e9db-4210-b534-527ed98e7333\") " pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:03:04.882382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.882290 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58-metrics-tls\") pod \"dns-default-9625v\" (UID: \"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58\") " pod="openshift-dns/dns-default-9625v" Oct 30 03:03:04.882382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.882297 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc68d015-081c-41bc-b7ec-2443b5693c76-metrics-certs\") pod \"network-metrics-daemon-qxx87\" (UID: \"cc68d015-081c-41bc-b7ec-2443b5693c76\") " pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:03:04.907445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.907425 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:04.927004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.926987 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qxx87" Oct 30 03:03:04.932515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.932500 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-9625v" Oct 30 03:03:04.932584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:04.932505 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cp2d8" Oct 30 03:03:05.106462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.106433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:05.228799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.228771 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qxx87"] Oct 30 03:03:05.232816 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:05.232786 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc68d015_081c_41bc_b7ec_2443b5693c76.slice/crio-5ebdd1910435115a4c60109f563cda3a52bfa2918e10bfdc17df8edfe3e7f219 WatchSource:0}: Error finding container 5ebdd1910435115a4c60109f563cda3a52bfa2918e10bfdc17df8edfe3e7f219: Status 404 returned error can't find the container with id 5ebdd1910435115a4c60109f563cda3a52bfa2918e10bfdc17df8edfe3e7f219 Oct 30 03:03:05.234830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.234813 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:03:05.241132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.241086 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-9625v"] Oct 30 03:03:05.243710 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:05.243690 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63a8507_8ba7_45ac_b6a0_e0e00c7b1a58.slice/crio-740c903ebcc2cf57fb6ac9dfbd20a751dc807127d99ff189544c6d0a5ef44012 WatchSource:0}: Error finding container 740c903ebcc2cf57fb6ac9dfbd20a751dc807127d99ff189544c6d0a5ef44012: Status 404 returned error can't find the container with id 740c903ebcc2cf57fb6ac9dfbd20a751dc807127d99ff189544c6d0a5ef44012 Oct 30 03:03:05.262469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.262449 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cp2d8"] Oct 30 03:03:05.265700 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:05.265671 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3f2dca_e9db_4210_b534_527ed98e7333.slice/crio-011fb0e354b94bd692894a521473946a3093488913126638709b09f39da88816 WatchSource:0}: Error finding container 011fb0e354b94bd692894a521473946a3093488913126638709b09f39da88816: Status 404 returned error can't find the container with id 011fb0e354b94bd692894a521473946a3093488913126638709b09f39da88816 Oct 30 03:03:05.285193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.285169 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:05.394322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.394292 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:05.503392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.503365 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:05.599512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.599481 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:05.700350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.700312 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:05.802199 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:05.802117 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:06.109720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.109643 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:06.159416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.159382 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qxx87" event={"ID":"cc68d015-081c-41bc-b7ec-2443b5693c76","Type":"ContainerStarted","Data":"5ebdd1910435115a4c60109f563cda3a52bfa2918e10bfdc17df8edfe3e7f219"} Oct 30 03:03:06.161677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.161652 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp2d8" event={"ID":"2c3f2dca-e9db-4210-b534-527ed98e7333","Type":"ContainerStarted","Data":"fed017594dd2d9a26c07f9650015ed748d642a9327b6579ec554de3e23195566"} Oct 30 03:03:06.161842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.161725 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp2d8" event={"ID":"2c3f2dca-e9db-4210-b534-527ed98e7333","Type":"ContainerStarted","Data":"29cdda49e83da0e3a8d39b074467c896d3e8e795e26122084ea931f7774e18f4"} Oct 30 03:03:06.161842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.161773 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp2d8" event={"ID":"2c3f2dca-e9db-4210-b534-527ed98e7333","Type":"ContainerStarted","Data":"011fb0e354b94bd692894a521473946a3093488913126638709b09f39da88816"} Oct 30 03:03:06.162903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.162879 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9625v" event={"ID":"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58","Type":"ContainerStarted","Data":"740c903ebcc2cf57fb6ac9dfbd20a751dc807127d99ff189544c6d0a5ef44012"} Oct 30 03:03:06.400366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.400328 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:06.499765 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:06.499727 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:07.169332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:07.169296 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9625v" event={"ID":"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58","Type":"ContainerStarted","Data":"91449a172f7c24ce320bd8b90e05b4d35b8650a468d6df52ca4e21bcda179fdb"} Oct 30 03:03:07.169822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:07.169389 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-9625v" event={"ID":"c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58","Type":"ContainerStarted","Data":"dc8bea625489f1ec067de59066af74727b8f0d03b02392cc41da87c72165efc0"} Oct 30 03:03:07.169822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:07.169414 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-9625v" Oct 30 03:03:07.171369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:07.171339 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qxx87" event={"ID":"cc68d015-081c-41bc-b7ec-2443b5693c76","Type":"ContainerStarted","Data":"757cb40ae4bd76a4f703f3ef6daab2f7c300cee7ba965081a04ae94a3a923435"} Oct 30 03:03:07.171499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:07.171387 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qxx87" event={"ID":"cc68d015-081c-41bc-b7ec-2443b5693c76","Type":"ContainerStarted","Data":"9c4957485989561880b36f05a987b25883ebbb9cdd3c83e35b2c7f38c0e6ac6c"} Oct 30 03:03:07.310649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:07.310613 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:08.179228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:08.179192 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cp2d8" event={"ID":"2c3f2dca-e9db-4210-b534-527ed98e7333","Type":"ContainerStarted","Data":"9dc710bb5d1bd0eca3bc86d05e2a624e2cd4aeb9e20f80c1fd52bf7f92194a90"} Oct 30 03:03:09.102458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.102424 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8684848d94-c9w9v"] Oct 30 03:03:09.105399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.105380 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.106003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.105978 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:09.144426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.144404 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 03:03:09.154936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.154913 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 03:03:09.155009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.154960 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 03:03:09.155009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.155006 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wcx8m\"" Oct 30 03:03:09.155090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.155037 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 03:03:09.155687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.155670 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8684848d94-c9w9v"] Oct 30 03:03:09.211326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211295 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-image-registry-private-configuration\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.211656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211332 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-registry-certificates\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.211656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211357 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ggc\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-kube-api-access-26ggc\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.211656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211484 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-ca-trust-extracted\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.211656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-registry-tls\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.211656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211532 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-installation-pull-secrets\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.211656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211553 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-bound-sa-token\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.211656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.211605 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-trusted-ca\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.294312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.294284 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:09.312723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.312702 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-image-registry-private-configuration\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.312829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.312726 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-registry-certificates\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.312829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.312744 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26ggc\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-kube-api-access-26ggc\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.312829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.312782 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-ca-trust-extracted\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.312968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.312898 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-registry-tls\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.312968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.312934 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-installation-pull-secrets\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.313057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.312983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-bound-sa-token\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.313057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.313010 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-trusted-ca\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.313160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.313143 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-ca-trust-extracted\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.313918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.313890 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-registry-certificates\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.313918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.313911 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-trusted-ca\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.315408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.315388 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-image-registry-private-configuration\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.315557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.315537 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-installation-pull-secrets\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.315616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.315537 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-registry-tls\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.331784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.331742 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ggc\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-kube-api-access-26ggc\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.331953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.331937 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a01652d2-9f9b-4ae6-9988-b27ee8d5952b-bound-sa-token\") pod \"image-registry-8684848d94-c9w9v\" (UID: \"a01652d2-9f9b-4ae6-9988-b27ee8d5952b\") " pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.396614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.396556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:09.416250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.416232 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:09.736912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:09.736839 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8684848d94-c9w9v"] Oct 30 03:03:09.739730 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:09.739697 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01652d2_9f9b_4ae6_9988_b27ee8d5952b.slice/crio-0a6fe0c7e83ead8bfeb524c2f786b4ff9748ab737cb742eb2799e1f49e15c022 WatchSource:0}: Error finding container 0a6fe0c7e83ead8bfeb524c2f786b4ff9748ab737cb742eb2799e1f49e15c022: Status 404 returned error can't find the container with id 0a6fe0c7e83ead8bfeb524c2f786b4ff9748ab737cb742eb2799e1f49e15c022 Oct 30 03:03:10.189787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:10.189735 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8684848d94-c9w9v" event={"ID":"a01652d2-9f9b-4ae6-9988-b27ee8d5952b","Type":"ContainerStarted","Data":"1cacbb13f253a5a72c0772a4cede35c05a357d35349fda69cb96c767addc7500"} Oct 30 03:03:10.189995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:10.189844 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8684848d94-c9w9v" event={"ID":"a01652d2-9f9b-4ae6-9988-b27ee8d5952b","Type":"ContainerStarted","Data":"0a6fe0c7e83ead8bfeb524c2f786b4ff9748ab737cb742eb2799e1f49e15c022"} Oct 30 03:03:10.189995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:10.189907 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:10.231731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:10.231634 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8684848d94-c9w9v" podStartSLOduration=1.2316206379999999 podStartE2EDuration="1.231620638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:10.231070998 +0000 UTC m=+14.233424196" watchObservedRunningTime="2026-10-30 03:03:10.231620638 +0000 UTC m=+14.233973835" Oct 30 03:03:12.442614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.442580 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:12.552564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.552539 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:12.560045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.560017 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c"] Oct 30 03:03:12.564695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.564680 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.588981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.588961 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 03:03:12.589326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.589311 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:03:12.589420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.589314 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:03:12.589420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.589368 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 03:03:12.589490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.589423 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lhjxf\"" Oct 30 03:03:12.589490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.589475 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 03:03:12.600963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.600942 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c"] Oct 30 03:03:12.736704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.736633 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f33aee02-cbf6-48d8-b56d-3b8895651792-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.736704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.736665 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7kk\" (UniqueName: \"kubernetes.io/projected/f33aee02-cbf6-48d8-b56d-3b8895651792-kube-api-access-ml7kk\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.736704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.736683 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f33aee02-cbf6-48d8-b56d-3b8895651792-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.736944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.736779 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f33aee02-cbf6-48d8-b56d-3b8895651792-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.795333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.795289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:12.837368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.837342 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f33aee02-cbf6-48d8-b56d-3b8895651792-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.837490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.837388 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f33aee02-cbf6-48d8-b56d-3b8895651792-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.837490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.837406 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7kk\" (UniqueName: \"kubernetes.io/projected/f33aee02-cbf6-48d8-b56d-3b8895651792-kube-api-access-ml7kk\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.837490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.837426 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f33aee02-cbf6-48d8-b56d-3b8895651792-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.838075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.838040 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f33aee02-cbf6-48d8-b56d-3b8895651792-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.839747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.839722 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f33aee02-cbf6-48d8-b56d-3b8895651792-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.839929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.839910 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f33aee02-cbf6-48d8-b56d-3b8895651792-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.856406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.856382 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7kk\" (UniqueName: \"kubernetes.io/projected/f33aee02-cbf6-48d8-b56d-3b8895651792-kube-api-access-ml7kk\") pod \"prometheus-operator-9cd89b74c-bjg6c\" (UID: \"f33aee02-cbf6-48d8-b56d-3b8895651792\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:12.877378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:12.877353 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" Oct 30 03:03:13.172746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:13.172720 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c"] Oct 30 03:03:13.175554 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:13.175523 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33aee02_cbf6_48d8_b56d_3b8895651792.slice/crio-374e29250ad0077807a91329b503ef5a86a696c61a93f1e74265e39b9205638d WatchSource:0}: Error finding container 374e29250ad0077807a91329b503ef5a86a696c61a93f1e74265e39b9205638d: Status 404 returned error can't find the container with id 374e29250ad0077807a91329b503ef5a86a696c61a93f1e74265e39b9205638d Oct 30 03:03:13.199873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:13.199841 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" event={"ID":"f33aee02-cbf6-48d8-b56d-3b8895651792","Type":"ContainerStarted","Data":"374e29250ad0077807a91329b503ef5a86a696c61a93f1e74265e39b9205638d"} Oct 30 03:03:15.210083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:15.210031 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" event={"ID":"f33aee02-cbf6-48d8-b56d-3b8895651792","Type":"ContainerStarted","Data":"b2eb01844afaedf0cb667c1d2f22bb73156b1e2e1accf1f25acd5e6136cf52de"} Oct 30 03:03:15.210593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:15.210104 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" event={"ID":"f33aee02-cbf6-48d8-b56d-3b8895651792","Type":"ContainerStarted","Data":"6c94abe8d807e8959feb25687643a717180c037aa54091ed3ad8f34177323607"} Oct 30 03:03:15.261643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:15.261578 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-bjg6c" podStartSLOduration=1.981701178 podStartE2EDuration="3.261561937s" totalImagesPullingTime="1.279860759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:15.259894915 +0000 UTC m=+19.262248111" watchObservedRunningTime="2026-10-30 03:03:15.261561937 +0000 UTC m=+19.263915135" Oct 30 03:03:15.288761 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:15.288732 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:16.945222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:16.945189 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:17.084767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.084723 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:17.182211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.182191 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-9625v" Oct 30 03:03:17.387239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.387201 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:17.540404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.540372 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:17.687909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.687834 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:17.781180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.779143 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z8zv7"] Oct 30 03:03:17.785082 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.785059 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.801706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.801678 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:17.855662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.855633 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 03:03:17.855887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.855873 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5lzhh\"" Oct 30 03:03:17.855998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.855986 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 03:03:17.856534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.856517 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 03:03:17.869957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.869935 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v9vj\" (UniqueName: \"kubernetes.io/projected/cd5d7d80-b047-4446-ad12-945bca88164b-kube-api-access-5v9vj\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.869969 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-tls\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.869988 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.870008 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-sys\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.870060 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-root\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.870120 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd5d7d80-b047-4446-ad12-945bca88164b-metrics-client-ca\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.870159 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-textfile\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.870195 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.870286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.870213 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-wtmp\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.905880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.905856 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:17.971318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971248 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-sys\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971284 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-root\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971312 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd5d7d80-b047-4446-ad12-945bca88164b-metrics-client-ca\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971345 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-textfile\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-sys\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971372 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-root\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971401 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971430 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-wtmp\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971498 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v9vj\" (UniqueName: \"kubernetes.io/projected/cd5d7d80-b047-4446-ad12-945bca88164b-kube-api-access-5v9vj\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971546 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-tls\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971574 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971592 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-wtmp\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.971857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.971836 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-textfile\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.972234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.972019 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd5d7d80-b047-4446-ad12-945bca88164b-metrics-client-ca\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.972234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.972186 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.974106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.974087 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-tls\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.974176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.974115 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd5d7d80-b047-4446-ad12-945bca88164b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:17.989095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:17.989073 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v9vj\" (UniqueName: \"kubernetes.io/projected/cd5d7d80-b047-4446-ad12-945bca88164b-kube-api-access-5v9vj\") pod \"node-exporter-z8zv7\" (UID: \"cd5d7d80-b047-4446-ad12-945bca88164b\") " pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:18.018376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:18.018340 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:18.100044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:18.100014 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8zv7" Oct 30 03:03:18.111138 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:18.111110 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5d7d80_b047_4446_ad12_945bca88164b.slice/crio-8852f22fbf5071f8ddbe5ffd9132be05ee02d31d551bc102e00ca1e5e9dd4ead WatchSource:0}: Error finding container 8852f22fbf5071f8ddbe5ffd9132be05ee02d31d551bc102e00ca1e5e9dd4ead: Status 404 returned error can't find the container with id 8852f22fbf5071f8ddbe5ffd9132be05ee02d31d551bc102e00ca1e5e9dd4ead Oct 30 03:03:18.219888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:18.219856 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8zv7" event={"ID":"cd5d7d80-b047-4446-ad12-945bca88164b","Type":"ContainerStarted","Data":"8852f22fbf5071f8ddbe5ffd9132be05ee02d31d551bc102e00ca1e5e9dd4ead"} Oct 30 03:03:18.339879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:18.339850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:18.490916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:18.490888 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:18.583348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:18.583315 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:18.939885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:18.939853 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j2q75" Oct 30 03:03:19.225357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:19.225284 4442 generic.go:356] "Generic (PLEG): container finished" podID="cd5d7d80-b047-4446-ad12-945bca88164b" containerID="b77491b64b2d156cbc7bbe16b0128ee57b74105834afa16cfae74e09644b0d20" exitCode=0 Oct 30 03:03:19.225357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:19.225344 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8zv7" event={"ID":"cd5d7d80-b047-4446-ad12-945bca88164b","Type":"ContainerDied","Data":"b77491b64b2d156cbc7bbe16b0128ee57b74105834afa16cfae74e09644b0d20"} Oct 30 03:03:20.230687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:20.230655 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8zv7" event={"ID":"cd5d7d80-b047-4446-ad12-945bca88164b","Type":"ContainerStarted","Data":"810b4986920400c08e2875ea23ef8c23cb911d36eba356295ff803f52405a6b6"} Oct 30 03:03:20.231067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:20.230703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8zv7" event={"ID":"cd5d7d80-b047-4446-ad12-945bca88164b","Type":"ContainerStarted","Data":"f716620287f7e83ee78289c4645050840891cb994d321d8046708e6d453c6230"} Oct 30 03:03:20.288871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:20.288844 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtbgx" Oct 30 03:03:20.292822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:20.292782 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z8zv7" podStartSLOduration=2.3293570949999998 podStartE2EDuration="3.292743526s" totalImagesPullingTime="889.041409ms" totalInitContainerRuntime="74.345022ms" isStatefulPod=true podCreationTimestamp="2026-10-30 03:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:20.281499135 +0000 UTC m=+24.283852333" watchObservedRunningTime="2026-10-30 03:03:20.292743526 +0000 UTC m=+24.295096716" Oct 30 03:03:20.443851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:20.443816 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:20.541287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:20.541256 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:20.899430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:20.899359 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:21.116895 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:21.116856 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:21.213747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:21.213672 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:21.397405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:21.397378 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:21.497931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:21.497866 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:22.021648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.021618 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:22.122177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.122148 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc"] Oct 30 03:03:22.126002 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.125986 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.145421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.145400 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 03:03:22.151294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.151230 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:22.152683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.152667 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-wxdgt\"" Oct 30 03:03:22.153772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.153731 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc"] Oct 30 03:03:22.179599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.179576 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-db4fc75f6-lzwkl"] Oct 30 03:03:22.182708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.182694 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.199551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.199531 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 03:03:22.204547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.204525 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-db4fc75f6-lzwkl"] Oct 30 03:03:22.242938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.242904 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 03:03:22.243080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.243057 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 03:03:22.243147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.243131 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7s8b2466o1f6q\"" Oct 30 03:03:22.243240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.243225 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-csxqg\"" Oct 30 03:03:22.262224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.262199 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:22.301226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301175 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 03:03:22.301332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301316 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6c174-f353-40f4-9e1b-f4719ea7b024-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.301372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301342 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-client-ca-bundle\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.301372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301363 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzfs\" (UniqueName: \"kubernetes.io/projected/3ab9f131-32d6-49d4-8c83-60938c842a3f-kube-api-access-5mzfs\") pod \"monitoring-plugin-77f4f76594-v8mhc\" (UID: \"3ab9f131-32d6-49d4-8c83-60938c842a3f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.301445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301427 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ab9f131-32d6-49d4-8c83-60938c842a3f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-v8mhc\" (UID: \"3ab9f131-32d6-49d4-8c83-60938c842a3f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.301508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301496 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-secret-metrics-server-tls\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.301544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301532 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-secret-metrics-server-client-certs\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.301576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301552 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98c6c174-f353-40f4-9e1b-f4719ea7b024-metrics-server-audit-profiles\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.301610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301574 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tggn6\" (UniqueName: \"kubernetes.io/projected/98c6c174-f353-40f4-9e1b-f4719ea7b024-kube-api-access-tggn6\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.301643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.301614 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98c6c174-f353-40f4-9e1b-f4719ea7b024-audit-log\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402282 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ab9f131-32d6-49d4-8c83-60938c842a3f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-v8mhc\" (UID: \"3ab9f131-32d6-49d4-8c83-60938c842a3f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.402316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402318 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-secret-metrics-server-tls\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402341 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-secret-metrics-server-client-certs\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402361 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98c6c174-f353-40f4-9e1b-f4719ea7b024-metrics-server-audit-profiles\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402377 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tggn6\" (UniqueName: \"kubernetes.io/projected/98c6c174-f353-40f4-9e1b-f4719ea7b024-kube-api-access-tggn6\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402427 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98c6c174-f353-40f4-9e1b-f4719ea7b024-audit-log\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402485 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6c174-f353-40f4-9e1b-f4719ea7b024-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402514 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-client-ca-bundle\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.402893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzfs\" (UniqueName: \"kubernetes.io/projected/3ab9f131-32d6-49d4-8c83-60938c842a3f-kube-api-access-5mzfs\") pod \"monitoring-plugin-77f4f76594-v8mhc\" (UID: \"3ab9f131-32d6-49d4-8c83-60938c842a3f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.403285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.402933 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/98c6c174-f353-40f4-9e1b-f4719ea7b024-audit-log\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.403353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.403292 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98c6c174-f353-40f4-9e1b-f4719ea7b024-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.403767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.403714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/98c6c174-f353-40f4-9e1b-f4719ea7b024-metrics-server-audit-profiles\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.405165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.405138 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-secret-metrics-server-client-certs\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.405290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.405185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ab9f131-32d6-49d4-8c83-60938c842a3f-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-v8mhc\" (UID: \"3ab9f131-32d6-49d4-8c83-60938c842a3f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.405513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.405493 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-client-ca-bundle\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.405608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.405591 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/98c6c174-f353-40f4-9e1b-f4719ea7b024-secret-metrics-server-tls\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.422248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.422228 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tggn6\" (UniqueName: \"kubernetes.io/projected/98c6c174-f353-40f4-9e1b-f4719ea7b024-kube-api-access-tggn6\") pod \"metrics-server-db4fc75f6-lzwkl\" (UID: \"98c6c174-f353-40f4-9e1b-f4719ea7b024\") " pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.423741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.423719 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzfs\" (UniqueName: \"kubernetes.io/projected/3ab9f131-32d6-49d4-8c83-60938c842a3f-kube-api-access-5mzfs\") pod \"monitoring-plugin-77f4f76594-v8mhc\" (UID: \"3ab9f131-32d6-49d4-8c83-60938c842a3f\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.436582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.436558 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:22.495271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.494130 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:22.730682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.730599 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc"] Oct 30 03:03:22.733159 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:22.733134 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab9f131_32d6_49d4_8c83_60938c842a3f.slice/crio-005cb5d0b4f1384c405c5afab8a50939619be918b8d5c09dd96c259f340c7fd2 WatchSource:0}: Error finding container 005cb5d0b4f1384c405c5afab8a50939619be918b8d5c09dd96c259f340c7fd2: Status 404 returned error can't find the container with id 005cb5d0b4f1384c405c5afab8a50939619be918b8d5c09dd96c259f340c7fd2 Oct 30 03:03:22.789930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:22.789892 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-db4fc75f6-lzwkl"] Oct 30 03:03:22.793274 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:03:22.793247 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c6c174_f353_40f4_9e1b_f4719ea7b024.slice/crio-686113e61bba3c01ce031b846b88c0d099850a9e6982bae334571d5a06a7d163 WatchSource:0}: Error finding container 686113e61bba3c01ce031b846b88c0d099850a9e6982bae334571d5a06a7d163: Status 404 returned error can't find the container with id 686113e61bba3c01ce031b846b88c0d099850a9e6982bae334571d5a06a7d163 Oct 30 03:03:23.244460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:23.244421 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" event={"ID":"3ab9f131-32d6-49d4-8c83-60938c842a3f","Type":"ContainerStarted","Data":"005cb5d0b4f1384c405c5afab8a50939619be918b8d5c09dd96c259f340c7fd2"} Oct 30 03:03:23.245851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:23.245810 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" event={"ID":"98c6c174-f353-40f4-9e1b-f4719ea7b024","Type":"ContainerStarted","Data":"686113e61bba3c01ce031b846b88c0d099850a9e6982bae334571d5a06a7d163"} Oct 30 03:03:24.123318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:24.123231 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:24.237380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:24.237347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:24.859444 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:24.859406 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:03:25.253118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:25.253078 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" event={"ID":"98c6c174-f353-40f4-9e1b-f4719ea7b024","Type":"ContainerStarted","Data":"01a0caf103cca591997759e6ea38d0aba2b0a26d76ceaf3701aad2c67a970db8"} Oct 30 03:03:25.254646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:25.254573 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" event={"ID":"3ab9f131-32d6-49d4-8c83-60938c842a3f","Type":"ContainerStarted","Data":"8dc1b3018133dde70e8d15538923764314fec722948279c782f201e282ef5994"} Oct 30 03:03:25.254793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:25.254676 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:25.259677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:25.259659 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" Oct 30 03:03:25.292631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:25.292590 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" podStartSLOduration=1.802743253 podStartE2EDuration="3.292578552s" totalImagesPullingTime="1.489835299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:25.2916455 +0000 UTC m=+29.293998697" watchObservedRunningTime="2026-10-30 03:03:25.292578552 +0000 UTC m=+29.294931739" Oct 30 03:03:25.329769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:25.329586 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-v8mhc" podStartSLOduration=1.880857197 podStartE2EDuration="3.32957271s" totalImagesPullingTime="1.448715513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:25.329251887 +0000 UTC m=+29.331605085" watchObservedRunningTime="2026-10-30 03:03:25.32957271 +0000 UTC m=+29.331925908" Oct 30 03:03:28.510915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:28.510881 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6szch" Oct 30 03:03:30.147981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:30.147937 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:30.234701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:30.234667 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:31.197638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:31.197610 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8684848d94-c9w9v" Oct 30 03:03:31.376666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:31.376636 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:31.461734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:31.461670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:31.549882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:31.549847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:32.709057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:32.709022 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:32.797451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:32.797415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:34.333212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:34.333182 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:34.437259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:34.437223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:34.730883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:34.730807 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:34.830064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:34.830040 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:34.930052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:34.930027 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:35.013845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:35.013814 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:35.382078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:35.382002 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:35.463092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:35.463065 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:36.398501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:36.398468 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:36.505073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:36.505034 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:42.024292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:42.024218 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:42.144767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:42.144709 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:42.495172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:42.495098 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:42.495172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:42.495159 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:03:44.470009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:44.469972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:44.558100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:44.558074 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:44.988828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:44.988792 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:45.100126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:45.100096 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:45.181444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:03:45.181416 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:03:54.859237 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:03:54.859198 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:04:02.500146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:02.500112 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:04:02.503890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:02.503872 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-db4fc75f6-lzwkl" Oct 30 03:04:04.360925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:04.360894 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:06.397380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:06.397349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:06.480760 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:06.480728 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:13.998536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:13.998501 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:14.087427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:14.087399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:23.818236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:23.818199 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:23.917373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:23.917332 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:24.142509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:24.142428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:24.243095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:24.243062 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:24.858574 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:04:24.858541 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:04:26.530465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:26.530425 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:26.617790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:26.617763 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:36.399960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:36.399923 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:36.501444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:36.501408 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:36.592631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:36.592599 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:42.139535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:42.139503 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:51.609319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:04:51.609284 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:04:54.859002 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:04:54.858965 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:05:24.860010 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:05:24.859928 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:05:35.837337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:05:35.837304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:05:35.940676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:05:35.940644 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:05:39.836767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:05:39.836715 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:05:39.932289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:05:39.932263 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:05:42.249796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:05:42.249766 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:05:54.859158 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:05:54.859120 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:06:12.943433 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:12.943393 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:24.437701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:24.437664 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:24.858920 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:06:24.858881 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:06:25.312694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.312598 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84"] Oct 30 03:06:25.315762 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.315737 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.341086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.341064 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-h889m\"" Oct 30 03:06:25.341248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.341141 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 03:06:25.341248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.341181 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 03:06:25.342455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.342423 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84"] Oct 30 03:06:25.344701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.344681 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:25.402343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.402319 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl"] Oct 30 03:06:25.405341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.405327 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.419882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.419861 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.419992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.419891 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.419992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.419911 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wzc\" (UniqueName: \"kubernetes.io/projected/e814b55f-de90-4fd0-9589-4da037389a22-kube-api-access-65wzc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.419992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.419934 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dx6\" (UniqueName: \"kubernetes.io/projected/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-kube-api-access-m8dx6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.420096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.420056 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.420136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.420098 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.424681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.424658 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl"] Oct 30 03:06:25.430287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.430269 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-qj5c5\"" Oct 30 03:06:25.520720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.520694 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.521201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.520722 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.521201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.520743 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wzc\" (UniqueName: \"kubernetes.io/projected/e814b55f-de90-4fd0-9589-4da037389a22-kube-api-access-65wzc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.521201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.520776 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dx6\" (UniqueName: \"kubernetes.io/projected/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-kube-api-access-m8dx6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.521201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.520796 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.521201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.520839 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.521201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.521193 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.521420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.521202 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.521420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.521208 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.521420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.521207 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.539338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.539313 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dx6\" (UniqueName: \"kubernetes.io/projected/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-kube-api-access-m8dx6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.539338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.539330 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wzc\" (UniqueName: \"kubernetes.io/projected/e814b55f-de90-4fd0-9589-4da037389a22-kube-api-access-65wzc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.630678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.630624 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:25.719832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.719800 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:25.764131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.764100 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf"] Oct 30 03:06:25.767727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.767704 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.772848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.772784 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:25.781865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.781691 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 03:06:25.781865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.781788 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 03:06:25.781865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.781840 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 03:06:25.794791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.794702 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf"] Oct 30 03:06:25.822408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.822384 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.822532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.822411 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.822532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.822431 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.822532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.822498 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ebce7e-462f-4bc1-8d26-af92d31e5846-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.822668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.822587 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.822668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.822628 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.822668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.822649 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcrhq\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-kube-api-access-jcrhq\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.861516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.861487 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:25.870861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.870829 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 03:06:25.870927 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.870897 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 03:06:25.870927 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.870908 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 03:06:25.870992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.870965 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ttlms\"" Oct 30 03:06:25.923094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.922982 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84"] Oct 30 03:06:25.923170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.923149 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.923237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.923221 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.923292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.923264 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcrhq\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-kube-api-access-jcrhq\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.923346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.923292 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.923346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.923318 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.923447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.923342 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.923447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.923383 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ebce7e-462f-4bc1-8d26-af92d31e5846-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.924091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.924069 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.925812 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:06:25.925786 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode814b55f_de90_4fd0_9589_4da037389a22.slice/crio-1c3156ea673e0197bc3d1d74b3100a33e26f560a4ca34aaf6836fe1b9efa2703 WatchSource:0}: Error finding container 1c3156ea673e0197bc3d1d74b3100a33e26f560a4ca34aaf6836fe1b9efa2703: Status 404 returned error can't find the container with id 1c3156ea673e0197bc3d1d74b3100a33e26f560a4ca34aaf6836fe1b9efa2703 Oct 30 03:06:25.926031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.926009 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.926142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.926124 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ebce7e-462f-4bc1-8d26-af92d31e5846-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.926206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.926173 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.926265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.926227 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.944159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.944129 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcrhq\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-kube-api-access-jcrhq\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:25.944387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:25.944372 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-l4skf\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:26.018909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:26.018882 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl"] Oct 30 03:06:26.021396 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:06:26.021367 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3c3866_ab4f_4aca_8562_7f2f57e38a12.slice/crio-6fb60de87979a11f1a4d59baea15da3a5e93ff30dfb9e09c075bc132ed7fb144 WatchSource:0}: Error finding container 6fb60de87979a11f1a4d59baea15da3a5e93ff30dfb9e09c075bc132ed7fb144: Status 404 returned error can't find the container with id 6fb60de87979a11f1a4d59baea15da3a5e93ff30dfb9e09c075bc132ed7fb144 Oct 30 03:06:26.085929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:26.085893 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:26.393513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:26.393474 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf"] Oct 30 03:06:26.397039 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:06:26.397008 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ebce7e_462f_4bc1_8d26_af92d31e5846.slice/crio-5683fd445462a21eca401d0f190a658842f5dca28a338bd80710cf13d9434b15 WatchSource:0}: Error finding container 5683fd445462a21eca401d0f190a658842f5dca28a338bd80710cf13d9434b15: Status 404 returned error can't find the container with id 5683fd445462a21eca401d0f190a658842f5dca28a338bd80710cf13d9434b15 Oct 30 03:06:26.812000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:26.811956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:26.825138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:26.825098 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" event={"ID":"33ebce7e-462f-4bc1-8d26-af92d31e5846","Type":"ContainerStarted","Data":"5683fd445462a21eca401d0f190a658842f5dca28a338bd80710cf13d9434b15"} Oct 30 03:06:26.826681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:26.826658 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" event={"ID":"ba3c3866-ab4f-4aca-8562-7f2f57e38a12","Type":"ContainerStarted","Data":"6fb60de87979a11f1a4d59baea15da3a5e93ff30dfb9e09c075bc132ed7fb144"} Oct 30 03:06:26.829288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:26.829237 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" event={"ID":"e814b55f-de90-4fd0-9589-4da037389a22","Type":"ContainerStarted","Data":"1c3156ea673e0197bc3d1d74b3100a33e26f560a4ca34aaf6836fe1b9efa2703"} Oct 30 03:06:29.875687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:29.875644 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:06:29.876098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:29.875732 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:06:30.600835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:30.600776 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:30.700923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:30.700861 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:30.848222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:30.848181 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" event={"ID":"33ebce7e-462f-4bc1-8d26-af92d31e5846","Type":"ContainerStarted","Data":"de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c"} Oct 30 03:06:30.848469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:30.848443 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:30.850404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:30.850370 4442 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-l4skf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:06:30.850553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:30.850448 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" podUID="33ebce7e-462f-4bc1-8d26-af92d31e5846" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:06:30.912679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:30.912569 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" podStartSLOduration=2.43711768 podStartE2EDuration="5.912552871s" totalImagesPullingTime="3.475435191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:30.900990249 +0000 UTC m=+214.903343445" watchObservedRunningTime="2026-10-30 03:06:30.912552871 +0000 UTC m=+214.914906067" Oct 30 03:06:31.457732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:31.457699 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:31.564364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:31.564330 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:31.663444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:31.663372 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:31.852657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:31.852626 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:06:32.856787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:32.856736 4442 generic.go:356] "Generic (PLEG): container finished" podID="e814b55f-de90-4fd0-9589-4da037389a22" containerID="8c14e5b8dae79ef3a4ef6d06ee9cf9236a63901284f860b6314733b6f5c9d430" exitCode=0 Oct 30 03:06:32.857133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:32.856786 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" event={"ID":"e814b55f-de90-4fd0-9589-4da037389a22","Type":"ContainerDied","Data":"8c14e5b8dae79ef3a4ef6d06ee9cf9236a63901284f860b6314733b6f5c9d430"} Oct 30 03:06:33.861175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:33.861141 4442 generic.go:356] "Generic (PLEG): container finished" podID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerID="f681a09cd6fa3ff973d280c5b5c6dd7a1c4ce1cc2c83763660bc4badc2514414" exitCode=0 Oct 30 03:06:33.861636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:33.861232 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" event={"ID":"ba3c3866-ab4f-4aca-8562-7f2f57e38a12","Type":"ContainerDied","Data":"f681a09cd6fa3ff973d280c5b5c6dd7a1c4ce1cc2c83763660bc4badc2514414"} Oct 30 03:06:33.863233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:33.863213 4442 generic.go:356] "Generic (PLEG): container finished" podID="e814b55f-de90-4fd0-9589-4da037389a22" containerID="9bd203348bdcfc9d47b1dc9d6d4ffc65c9026abbfd397e851223f010ebc25dff" exitCode=0 Oct 30 03:06:33.863328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:33.863246 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" event={"ID":"e814b55f-de90-4fd0-9589-4da037389a22","Type":"ContainerDied","Data":"9bd203348bdcfc9d47b1dc9d6d4ffc65c9026abbfd397e851223f010ebc25dff"} Oct 30 03:06:35.016908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:35.016834 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:35.873988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:35.873953 4442 generic.go:356] "Generic (PLEG): container finished" podID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerID="dda17bbb67d681e7e1afd4de24f88542bc2fa8db62a9be09084703976e883511" exitCode=0 Oct 30 03:06:35.874142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:35.874004 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" event={"ID":"ba3c3866-ab4f-4aca-8562-7f2f57e38a12","Type":"ContainerDied","Data":"dda17bbb67d681e7e1afd4de24f88542bc2fa8db62a9be09084703976e883511"} Oct 30 03:06:40.896254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:40.896211 4442 generic.go:356] "Generic (PLEG): container finished" podID="e814b55f-de90-4fd0-9589-4da037389a22" containerID="d440557efe5a930e05286e992dafdb9c79cd73c662b2856a1a55b7a20d3cf550" exitCode=0 Oct 30 03:06:40.896744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:40.896305 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" event={"ID":"e814b55f-de90-4fd0-9589-4da037389a22","Type":"ContainerDied","Data":"d440557efe5a930e05286e992dafdb9c79cd73c662b2856a1a55b7a20d3cf550"} Oct 30 03:06:42.021651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.021625 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:42.064095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.064076 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-util\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-util\") pod \"e814b55f-de90-4fd0-9589-4da037389a22\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " Oct 30 03:06:42.064184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.064107 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-bundle\") pod \"e814b55f-de90-4fd0-9589-4da037389a22\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " Oct 30 03:06:42.064184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.064152 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e814b55f-de90-4fd0-9589-4da037389a22-kube-api-access-65wzc\" (UniqueName: \"kubernetes.io/projected/e814b55f-de90-4fd0-9589-4da037389a22-kube-api-access-65wzc\") pod \"e814b55f-de90-4fd0-9589-4da037389a22\" (UID: \"e814b55f-de90-4fd0-9589-4da037389a22\") " Oct 30 03:06:42.064673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.064603 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-bundle" pod "e814b55f-de90-4fd0-9589-4da037389a22" (UID: "e814b55f-de90-4fd0-9589-4da037389a22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:06:42.066465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.066437 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e814b55f-de90-4fd0-9589-4da037389a22-kube-api-access-65wzc" pod "e814b55f-de90-4fd0-9589-4da037389a22" (UID: "e814b55f-de90-4fd0-9589-4da037389a22"). InnerVolumeSpecName "kube-api-access-65wzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:06:42.068547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.068519 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-util" pod "e814b55f-de90-4fd0-9589-4da037389a22" (UID: "e814b55f-de90-4fd0-9589-4da037389a22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:06:42.165039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.164969 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-util\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:06:42.165039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.164994 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e814b55f-de90-4fd0-9589-4da037389a22-bundle\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:06:42.165039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.165003 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65wzc\" (UniqueName: \"kubernetes.io/projected/e814b55f-de90-4fd0-9589-4da037389a22-kube-api-access-65wzc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:06:42.381660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.381623 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:06:42.904522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.904479 4442 generic.go:356] "Generic (PLEG): container finished" podID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerID="5a4f255d5dfb05cb7a4c323d194f9688f1f5297013f65d4dd63e665248e78a5c" exitCode=0 Oct 30 03:06:42.904684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.904561 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" event={"ID":"ba3c3866-ab4f-4aca-8562-7f2f57e38a12","Type":"ContainerDied","Data":"5a4f255d5dfb05cb7a4c323d194f9688f1f5297013f65d4dd63e665248e78a5c"} Oct 30 03:06:42.906336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.906313 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" event={"ID":"e814b55f-de90-4fd0-9589-4da037389a22","Type":"ContainerDied","Data":"1c3156ea673e0197bc3d1d74b3100a33e26f560a4ca34aaf6836fe1b9efa2703"} Oct 30 03:06:42.906469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.906344 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mtc84" Oct 30 03:06:42.906469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:42.906348 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c3156ea673e0197bc3d1d74b3100a33e26f560a4ca34aaf6836fe1b9efa2703" Oct 30 03:06:44.039533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.039512 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:44.182777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.182697 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-kube-api-access-m8dx6\" (UniqueName: \"kubernetes.io/projected/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-kube-api-access-m8dx6\") pod \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " Oct 30 03:06:44.182777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.182744 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-util\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-util\") pod \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " Oct 30 03:06:44.182959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.182786 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-bundle\") pod \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\" (UID: \"ba3c3866-ab4f-4aca-8562-7f2f57e38a12\") " Oct 30 03:06:44.183214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.183187 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-bundle" pod "ba3c3866-ab4f-4aca-8562-7f2f57e38a12" (UID: "ba3c3866-ab4f-4aca-8562-7f2f57e38a12"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:06:44.185021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.185003 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-kube-api-access-m8dx6" pod "ba3c3866-ab4f-4aca-8562-7f2f57e38a12" (UID: "ba3c3866-ab4f-4aca-8562-7f2f57e38a12"). InnerVolumeSpecName "kube-api-access-m8dx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:06:44.188249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.188223 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-util" pod "ba3c3866-ab4f-4aca-8562-7f2f57e38a12" (UID: "ba3c3866-ab4f-4aca-8562-7f2f57e38a12"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:06:44.284055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.284025 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dx6\" (UniqueName: \"kubernetes.io/projected/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-kube-api-access-m8dx6\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:06:44.284055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.284050 4442 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-util\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:06:44.284055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.284061 4442 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ba3c3866-ab4f-4aca-8562-7f2f57e38a12-bundle\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:06:44.914404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.914362 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" event={"ID":"ba3c3866-ab4f-4aca-8562-7f2f57e38a12","Type":"ContainerDied","Data":"6fb60de87979a11f1a4d59baea15da3a5e93ff30dfb9e09c075bc132ed7fb144"} Oct 30 03:06:44.914404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.914383 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64457fdl" Oct 30 03:06:44.914404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:06:44.914410 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fb60de87979a11f1a4d59baea15da3a5e93ff30dfb9e09c075bc132ed7fb144" Oct 30 03:06:54.859675 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:06:54.859629 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:07:03.220843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:03.220807 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:03.335822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:03.335794 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:03.874946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:03.874912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:04.051476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051444 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd"] Oct 30 03:07:04.051726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051676 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e814b55f-de90-4fd0-9589-4da037389a22" containerName="util" Oct 30 03:07:04.051726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051689 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e814b55f-de90-4fd0-9589-4da037389a22" containerName="util" Oct 30 03:07:04.051726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051697 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerName="extract" Oct 30 03:07:04.051726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051702 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerName="extract" Oct 30 03:07:04.051726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051714 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerName="util" Oct 30 03:07:04.051726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051720 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerName="util" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051730 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerName="pull" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051743 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerName="pull" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051762 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e814b55f-de90-4fd0-9589-4da037389a22" containerName="extract" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051766 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e814b55f-de90-4fd0-9589-4da037389a22" containerName="extract" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051786 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e814b55f-de90-4fd0-9589-4da037389a22" containerName="pull" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051790 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e814b55f-de90-4fd0-9589-4da037389a22" containerName="pull" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051840 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3c3866-ab4f-4aca-8562-7f2f57e38a12" containerName="extract" Oct 30 03:07:04.051980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.051855 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e814b55f-de90-4fd0-9589-4da037389a22" containerName="extract" Oct 30 03:07:04.055691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.055671 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.078804 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.078773 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd"] Oct 30 03:07:04.147650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.147583 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 03:07:04.147650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.147592 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 03:07:04.150065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.150039 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 03:07:04.150320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.150303 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 03:07:04.150409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.150304 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-2t4j2\"" Oct 30 03:07:04.150409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.150322 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 03:07:04.151060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.151042 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:04.223609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.223586 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f44dab0e-73f4-4b88-a734-5154801ae8f5-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.223999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.223623 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f44dab0e-73f4-4b88-a734-5154801ae8f5-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.223999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.223647 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zj42\" (UniqueName: \"kubernetes.io/projected/f44dab0e-73f4-4b88-a734-5154801ae8f5-kube-api-access-5zj42\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.223999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.223679 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.223999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.223892 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.223999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.223930 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.253426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.253402 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:04.324806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.324778 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zj42\" (UniqueName: \"kubernetes.io/projected/f44dab0e-73f4-4b88-a734-5154801ae8f5-kube-api-access-5zj42\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.324911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.324814 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.324911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.324844 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.324911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.324860 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.324911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.324891 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f44dab0e-73f4-4b88-a734-5154801ae8f5-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.325102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.324931 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f44dab0e-73f4-4b88-a734-5154801ae8f5-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.325394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.325371 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f44dab0e-73f4-4b88-a734-5154801ae8f5-manifests\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.327342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.327317 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f44dab0e-73f4-4b88-a734-5154801ae8f5-tmp\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.327568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.327550 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.327637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.327565 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.327674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.327630 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f44dab0e-73f4-4b88-a734-5154801ae8f5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.346253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.346229 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zj42\" (UniqueName: \"kubernetes.io/projected/f44dab0e-73f4-4b88-a734-5154801ae8f5-kube-api-access-5zj42\") pod \"opendatahub-operator-controller-manager-5c85558465-pdrsd\" (UID: \"f44dab0e-73f4-4b88-a734-5154801ae8f5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.346973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.346953 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:04.367614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.367587 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:04.447598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.447568 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:04.546773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.546721 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:04.740367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.739790 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd"] Oct 30 03:07:04.753391 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:07:04.753358 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44dab0e_73f4_4b88_a734_5154801ae8f5.slice/crio-bad9f7d250621c507cf68eaaa737926fcdea1ec769f8eb0f40630289db500316 WatchSource:0}: Error finding container bad9f7d250621c507cf68eaaa737926fcdea1ec769f8eb0f40630289db500316: Status 404 returned error can't find the container with id bad9f7d250621c507cf68eaaa737926fcdea1ec769f8eb0f40630289db500316 Oct 30 03:07:04.982588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:04.982550 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" event={"ID":"f44dab0e-73f4-4b88-a734-5154801ae8f5","Type":"ContainerStarted","Data":"bad9f7d250621c507cf68eaaa737926fcdea1ec769f8eb0f40630289db500316"} Oct 30 03:07:05.192832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:05.192798 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:05.288864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:05.288816 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:06.834244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:06.834205 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:06.928619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:06.928570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:07.995113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:07.995036 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" event={"ID":"f44dab0e-73f4-4b88-a734-5154801ae8f5","Type":"ContainerStarted","Data":"f8c0529fa7abe601c91597bea147734f264427e4f4cf86e62fb002aa41632705"} Oct 30 03:07:08.999683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:08.999649 4442 generic.go:356] "Generic (PLEG): container finished" podID="f44dab0e-73f4-4b88-a734-5154801ae8f5" containerID="f8c0529fa7abe601c91597bea147734f264427e4f4cf86e62fb002aa41632705" exitCode=0 Oct 30 03:07:09.000101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:08.999706 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" event={"ID":"f44dab0e-73f4-4b88-a734-5154801ae8f5","Type":"ContainerDied","Data":"f8c0529fa7abe601c91597bea147734f264427e4f4cf86e62fb002aa41632705"} Oct 30 03:07:10.005441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:10.005407 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" event={"ID":"f44dab0e-73f4-4b88-a734-5154801ae8f5","Type":"ContainerStarted","Data":"9895b962b76c4f0b85bb94375750110f0ea3935b9946e62192ba3cdf2a64765d"} Oct 30 03:07:10.005850 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:10.005532 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:10.095999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:10.095955 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" podStartSLOduration=3.547066766 podStartE2EDuration="7.09594337s" totalImagesPullingTime="3.090295977s" totalInitContainerRuntime="458.580627ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:10.084487268 +0000 UTC m=+254.086840464" watchObservedRunningTime="2026-10-30 03:07:10.09594337 +0000 UTC m=+254.098296566" Oct 30 03:07:10.410574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:10.410494 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:10.506801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:10.506773 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:11.213803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:11.213769 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:11.311796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:11.311768 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:12.534128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:12.534091 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:16.426320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:16.426286 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:21.011859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:21.011827 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5c85558465-pdrsd" Oct 30 03:07:21.245422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:21.245387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:21.789667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:21.789634 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:24.618130 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:24.618091 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:24.859705 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:07:24.859664 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:07:25.152588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:25.152559 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:25.400002 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:25.399965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:26.420243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:26.420178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:27.824990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:27.824954 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:29.670644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:29.670612 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:29.815571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:29.815526 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:30.188715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:30.188683 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:30.432532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:30.432497 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:31.413532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:31.413497 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:31.884214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:31.884172 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:32.115256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:32.115225 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:32.400711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:32.400679 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:32.571635 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:32.571588 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:32.840673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:32.840639 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:36.834560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:36.834526 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:36.965941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:36.965909 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:37.230227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.230157 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:37.500257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.500228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:37.506444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.506410 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8"] Oct 30 03:07:37.512722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.512704 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.530424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.530399 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8"] Oct 30 03:07:37.554366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.554336 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 03:07:37.554366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.554352 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 03:07:37.554366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.554336 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-krtxf\"" Oct 30 03:07:37.572521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.572492 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pljc\" (UniqueName: \"kubernetes.io/projected/b5b60c17-548f-41cc-9313-5836a6171c81-kube-api-access-2pljc\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.572626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.572528 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b60c17-548f-41cc-9313-5836a6171c81-tls-certs\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.572669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.572654 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b60c17-548f-41cc-9313-5836a6171c81-tmp\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.647238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.647212 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:37.673440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.673410 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pljc\" (UniqueName: \"kubernetes.io/projected/b5b60c17-548f-41cc-9313-5836a6171c81-kube-api-access-2pljc\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.673557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.673452 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b60c17-548f-41cc-9313-5836a6171c81-tls-certs\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.673557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.673528 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b60c17-548f-41cc-9313-5836a6171c81-tmp\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.675841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.675822 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5b60c17-548f-41cc-9313-5836a6171c81-tmp\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.675971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.675955 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5b60c17-548f-41cc-9313-5836a6171c81-tls-certs\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.732969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.732950 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pljc\" (UniqueName: \"kubernetes.io/projected/b5b60c17-548f-41cc-9313-5836a6171c81-kube-api-access-2pljc\") pod \"kube-auth-proxy-78ccd95bcf-bjqd8\" (UID: \"b5b60c17-548f-41cc-9313-5836a6171c81\") " pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.824702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.824647 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" Oct 30 03:07:37.879507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:37.879444 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:38.006780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:38.006720 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:38.136437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:38.136373 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8"] Oct 30 03:07:38.139694 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:07:38.139667 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b60c17_548f_41cc_9313_5836a6171c81.slice/crio-78075f2db3bdeafe55f1c849c19fb5cac80b6049e297135192a8fd13ad64848b WatchSource:0}: Error finding container 78075f2db3bdeafe55f1c849c19fb5cac80b6049e297135192a8fd13ad64848b: Status 404 returned error can't find the container with id 78075f2db3bdeafe55f1c849c19fb5cac80b6049e297135192a8fd13ad64848b Oct 30 03:07:38.415384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:38.415297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:38.525127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:38.525092 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:39.090387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:39.090347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:39.108289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:39.108260 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" event={"ID":"b5b60c17-548f-41cc-9313-5836a6171c81","Type":"ContainerStarted","Data":"78075f2db3bdeafe55f1c849c19fb5cac80b6049e297135192a8fd13ad64848b"} Oct 30 03:07:39.236858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:39.236816 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:39.427265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:39.427190 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:40.126096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.126064 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:40.244149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.244109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:40.361136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.361094 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:40.607677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.607645 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:07:40.610861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.610842 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:40.626215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.626198 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 03:07:40.636470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.636443 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 03:07:40.636470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.636452 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 03:07:40.636602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.636507 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-clchq\"" Oct 30 03:07:40.638105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.638088 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:07:40.698873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.698843 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvzvz\" (UniqueName: \"kubernetes.io/projected/874a2d48-0e3f-45c1-bce3-bfccb9d006b8-kube-api-access-fvzvz\") pod \"limitador-limitador-7c9b87d574-ndw77\" (UID: \"874a2d48-0e3f-45c1-bce3-bfccb9d006b8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:40.699017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.698910 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/874a2d48-0e3f-45c1-bce3-bfccb9d006b8-config-file\") pod \"limitador-limitador-7c9b87d574-ndw77\" (UID: \"874a2d48-0e3f-45c1-bce3-bfccb9d006b8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:40.704513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.704448 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:07:40.741391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.741369 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:40.799796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.799748 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/874a2d48-0e3f-45c1-bce3-bfccb9d006b8-config-file\") pod \"limitador-limitador-7c9b87d574-ndw77\" (UID: \"874a2d48-0e3f-45c1-bce3-bfccb9d006b8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:40.799919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.799848 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvzvz\" (UniqueName: \"kubernetes.io/projected/874a2d48-0e3f-45c1-bce3-bfccb9d006b8-kube-api-access-fvzvz\") pod \"limitador-limitador-7c9b87d574-ndw77\" (UID: \"874a2d48-0e3f-45c1-bce3-bfccb9d006b8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:40.800295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.800276 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/874a2d48-0e3f-45c1-bce3-bfccb9d006b8-config-file\") pod \"limitador-limitador-7c9b87d574-ndw77\" (UID: \"874a2d48-0e3f-45c1-bce3-bfccb9d006b8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:40.818543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.818525 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvzvz\" (UniqueName: \"kubernetes.io/projected/874a2d48-0e3f-45c1-bce3-bfccb9d006b8-kube-api-access-fvzvz\") pod \"limitador-limitador-7c9b87d574-ndw77\" (UID: \"874a2d48-0e3f-45c1-bce3-bfccb9d006b8\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:40.922489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:40.922464 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:41.117350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:41.117318 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" event={"ID":"b5b60c17-548f-41cc-9313-5836a6171c81","Type":"ContainerStarted","Data":"b92b0467e710910fdc5e433b1950755976016ae72d3e4d027aa76e4a7820103e"} Oct 30 03:07:41.161801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:41.161730 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78ccd95bcf-bjqd8" podStartSLOduration=1.852703445 podStartE2EDuration="4.161715968s" totalImagesPullingTime="2.309012523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:41.161713771 +0000 UTC m=+285.164066981" watchObservedRunningTime="2026-10-30 03:07:41.161715968 +0000 UTC m=+285.164069162" Oct 30 03:07:41.167326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:41.167302 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:41.216639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:41.216611 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:07:41.219298 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:07:41.219276 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874a2d48_0e3f_45c1_bce3_bfccb9d006b8.slice/crio-ea53025ff02a2a5bf8ad3868006495aa6703c5696031d0c6abfdf86bb0293d43 WatchSource:0}: Error finding container ea53025ff02a2a5bf8ad3868006495aa6703c5696031d0c6abfdf86bb0293d43: Status 404 returned error can't find the container with id ea53025ff02a2a5bf8ad3868006495aa6703c5696031d0c6abfdf86bb0293d43 Oct 30 03:07:42.122983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:42.122950 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" event={"ID":"874a2d48-0e3f-45c1-bce3-bfccb9d006b8","Type":"ContainerStarted","Data":"ea53025ff02a2a5bf8ad3868006495aa6703c5696031d0c6abfdf86bb0293d43"} Oct 30 03:07:42.550608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:42.550573 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:42.683814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:42.683787 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:44.131603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:44.131570 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" event={"ID":"874a2d48-0e3f-45c1-bce3-bfccb9d006b8","Type":"ContainerStarted","Data":"90fbac37b0ee7863695d773402b7af70217ca8c016a1692c7f8f92bc352d1839"} Oct 30 03:07:44.132037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:44.131701 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:44.169616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:44.169567 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" podStartSLOduration=2.254058206 podStartE2EDuration="4.169553685s" totalImagesPullingTime="1.915495479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:44.168515836 +0000 UTC m=+288.170869035" watchObservedRunningTime="2026-10-30 03:07:44.169553685 +0000 UTC m=+288.171906882" Oct 30 03:07:47.455213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:47.455178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:47.594093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:47.594026 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:47.713162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:47.713091 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:47.877584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:47.877556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:52.914023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:52.913987 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:53.290288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:53.290260 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:54.859524 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:07:54.859492 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:07:55.135950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:55.135882 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ndw77" Oct 30 03:07:55.772245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:55.772216 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:55.992633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:55.992604 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:56.111324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:56.111245 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:07:56.111519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:56.111504 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:07:59.076115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:59.076080 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:59.469584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:59.469506 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:07:59.752983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:07:59.752941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:00.353963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:00.353919 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:02.385795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:02.385736 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:02.595053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:02.595022 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:05.498636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:05.498577 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:07.746022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:07.745984 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:08.032261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:08.032228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:10.668031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:10.667952 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:10.847953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:10.847907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:16.120056 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:16.120023 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:16.277608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:16.277578 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:18.435294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.435258 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:18.516714 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.516683 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7cd5d64f84-4bwrn"] Oct 30 03:08:18.521902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.521879 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.545956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.545924 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8sjhl\"" Oct 30 03:08:18.546074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.546016 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 03:08:18.556813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.556792 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 03:08:18.556813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.556818 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 03:08:18.557012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.556937 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 03:08:18.557089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.557077 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 03:08:18.558524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.558500 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd5d64f84-4bwrn"] Oct 30 03:08:18.602910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.602883 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:18.679318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.679290 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-webhook-cert\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.679456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.679329 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-metrics-certs\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.679456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.679350 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5zd\" (UniqueName: \"kubernetes.io/projected/b35e59e6-59f9-43bb-8dba-293ac3a1c563-kube-api-access-2t5zd\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.780138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.780104 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-webhook-cert\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.780275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.780151 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-metrics-certs\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.780275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.780172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5zd\" (UniqueName: \"kubernetes.io/projected/b35e59e6-59f9-43bb-8dba-293ac3a1c563-kube-api-access-2t5zd\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.782852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.782835 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-metrics-certs\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.782919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.782874 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-webhook-cert\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.795186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.795164 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:18.797592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.797569 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5zd\" (UniqueName: \"kubernetes.io/projected/b35e59e6-59f9-43bb-8dba-293ac3a1c563-kube-api-access-2t5zd\") pod \"maas-controller-7cd5d64f84-4bwrn\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.832530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.832509 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:18.895935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:18.895907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:19.086292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.086210 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-679796c8cb-4dk5t"] Oct 30 03:08:19.088440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.088419 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.108831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.108808 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4dk5t"] Oct 30 03:08:19.129122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.127145 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7cd5d64f84-4bwrn"] Oct 30 03:08:19.130600 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:08:19.130575 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35e59e6_59f9_43bb_8dba_293ac3a1c563.slice/crio-8754a70a6b5917a6cafbace6b01a18a75618caa9064d800a7ac0be2f60c59831 WatchSource:0}: Error finding container 8754a70a6b5917a6cafbace6b01a18a75618caa9064d800a7ac0be2f60c59831: Status 404 returned error can't find the container with id 8754a70a6b5917a6cafbace6b01a18a75618caa9064d800a7ac0be2f60c59831 Oct 30 03:08:19.131874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.131858 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:08:19.183052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.183025 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-metrics-certs\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.183154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.183068 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-webhook-cert\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.183154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.183090 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbz5\" (UniqueName: \"kubernetes.io/projected/05381e85-4985-4f92-9d08-6cca60bf8059-kube-api-access-qxbz5\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.257390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.257355 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" event={"ID":"b35e59e6-59f9-43bb-8dba-293ac3a1c563","Type":"ContainerStarted","Data":"8754a70a6b5917a6cafbace6b01a18a75618caa9064d800a7ac0be2f60c59831"} Oct 30 03:08:19.283925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.283903 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-metrics-certs\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.284023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.283940 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-webhook-cert\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.284023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.283961 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbz5\" (UniqueName: \"kubernetes.io/projected/05381e85-4985-4f92-9d08-6cca60bf8059-kube-api-access-qxbz5\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.286426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.286402 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-webhook-cert\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.286512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.286447 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-metrics-certs\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.301998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.301971 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbz5\" (UniqueName: \"kubernetes.io/projected/05381e85-4985-4f92-9d08-6cca60bf8059-kube-api-access-qxbz5\") pod \"maas-controller-679796c8cb-4dk5t\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.399734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.399674 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:19.706962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:19.706716 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4dk5t"] Oct 30 03:08:19.710863 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:08:19.710836 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05381e85_4985_4f92_9d08_6cca60bf8059.slice/crio-fe1202396429b457c4b41fd85cbf7bb733c24d62756339eef05c2600c7ac71e5 WatchSource:0}: Error finding container fe1202396429b457c4b41fd85cbf7bb733c24d62756339eef05c2600c7ac71e5: Status 404 returned error can't find the container with id fe1202396429b457c4b41fd85cbf7bb733c24d62756339eef05c2600c7ac71e5 Oct 30 03:08:20.264703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:20.264634 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4dk5t" event={"ID":"05381e85-4985-4f92-9d08-6cca60bf8059","Type":"ContainerStarted","Data":"fe1202396429b457c4b41fd85cbf7bb733c24d62756339eef05c2600c7ac71e5"} Oct 30 03:08:22.064165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:22.064127 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:22.216242 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:22.216174 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:22.274183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:22.274155 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" event={"ID":"b35e59e6-59f9-43bb-8dba-293ac3a1c563","Type":"ContainerStarted","Data":"c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c"} Oct 30 03:08:22.274294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:22.274267 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:22.317083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:22.317026 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" podStartSLOduration=1.6323774420000001 podStartE2EDuration="4.317008892s" totalImagesPullingTime="2.68463145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:22.316772253 +0000 UTC m=+326.319125450" watchObservedRunningTime="2026-10-30 03:08:22.317008892 +0000 UTC m=+326.319362087" Oct 30 03:08:23.279339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:23.279293 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4dk5t" event={"ID":"05381e85-4985-4f92-9d08-6cca60bf8059","Type":"ContainerStarted","Data":"3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911"} Oct 30 03:08:23.279724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:23.279379 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:23.317923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:23.317885 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-679796c8cb-4dk5t" podStartSLOduration=1.78976488 podStartE2EDuration="4.317873598s" totalImagesPullingTime="2.528108718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:23.317599521 +0000 UTC m=+327.319952719" watchObservedRunningTime="2026-10-30 03:08:23.317873598 +0000 UTC m=+327.320226806" Oct 30 03:08:23.507603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:23.507566 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:23.604586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:23.604505 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:24.859087 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:08:24.859051 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:08:33.282105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:33.282066 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:34.285467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.285432 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:08:34.352534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.352502 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd5d64f84-4bwrn"] Oct 30 03:08:34.352851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.352823 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" podUID="b35e59e6-59f9-43bb-8dba-293ac3a1c563" containerName="manager" containerID="cri-o://c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c" gracePeriod=30 Oct 30 03:08:34.591675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.591654 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:34.705961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.705921 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-metrics-certs\") pod \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " Oct 30 03:08:34.705961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.705972 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-webhook-cert\") pod \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " Oct 30 03:08:34.706182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.706003 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35e59e6-59f9-43bb-8dba-293ac3a1c563-kube-api-access-2t5zd\" (UniqueName: \"kubernetes.io/projected/b35e59e6-59f9-43bb-8dba-293ac3a1c563-kube-api-access-2t5zd\") pod \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\" (UID: \"b35e59e6-59f9-43bb-8dba-293ac3a1c563\") " Oct 30 03:08:34.708273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.708244 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-metrics-certs" pod "b35e59e6-59f9-43bb-8dba-293ac3a1c563" (UID: "b35e59e6-59f9-43bb-8dba-293ac3a1c563"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:34.708273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.708263 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-webhook-cert" pod "b35e59e6-59f9-43bb-8dba-293ac3a1c563" (UID: "b35e59e6-59f9-43bb-8dba-293ac3a1c563"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:34.708409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.708326 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35e59e6-59f9-43bb-8dba-293ac3a1c563-kube-api-access-2t5zd" pod "b35e59e6-59f9-43bb-8dba-293ac3a1c563" (UID: "b35e59e6-59f9-43bb-8dba-293ac3a1c563"). InnerVolumeSpecName "kube-api-access-2t5zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:34.806924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.806889 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:34.806924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.806918 4442 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b35e59e6-59f9-43bb-8dba-293ac3a1c563-webhook-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:34.806924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:34.806929 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5zd\" (UniqueName: \"kubernetes.io/projected/b35e59e6-59f9-43bb-8dba-293ac3a1c563-kube-api-access-2t5zd\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:35.323556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.323516 4442 generic.go:356] "Generic (PLEG): container finished" podID="b35e59e6-59f9-43bb-8dba-293ac3a1c563" containerID="c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c" exitCode=0 Oct 30 03:08:35.324037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.323563 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" Oct 30 03:08:35.324037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.323571 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" event={"ID":"b35e59e6-59f9-43bb-8dba-293ac3a1c563","Type":"ContainerDied","Data":"c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c"} Oct 30 03:08:35.324037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.323623 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7cd5d64f84-4bwrn" event={"ID":"b35e59e6-59f9-43bb-8dba-293ac3a1c563","Type":"ContainerDied","Data":"8754a70a6b5917a6cafbace6b01a18a75618caa9064d800a7ac0be2f60c59831"} Oct 30 03:08:35.324037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.323646 4442 scope.go:122] "RemoveContainer" containerID="c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c" Oct 30 03:08:35.333059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.332985 4442 scope.go:122] "RemoveContainer" containerID="c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c" Oct 30 03:08:35.333361 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:08:35.333335 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c\": container with ID starting with c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c not found: ID does not exist" containerID="c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c" Oct 30 03:08:35.333441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.333377 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c"} err="failed to get container status \"c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c\": rpc error: code = NotFound desc = could not find container \"c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c\": container with ID starting with c3608ebb2f3a6f17a04a17df83e5657449e260177826dd5ac2b9282cfd09fc8c not found: ID does not exist" Oct 30 03:08:35.372841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.372813 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7cd5d64f84-4bwrn"] Oct 30 03:08:35.383591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.383573 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7cd5d64f84-4bwrn"] Oct 30 03:08:35.972664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.972622 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs"] Oct 30 03:08:35.972980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.972965 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b35e59e6-59f9-43bb-8dba-293ac3a1c563" containerName="manager" Oct 30 03:08:35.973032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.972982 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35e59e6-59f9-43bb-8dba-293ac3a1c563" containerName="manager" Oct 30 03:08:35.973066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.973058 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35e59e6-59f9-43bb-8dba-293ac3a1c563" containerName="manager" Oct 30 03:08:35.977374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:35.977360 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.011872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.011848 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs"] Oct 30 03:08:36.104013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.103988 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b35e59e6-59f9-43bb-8dba-293ac3a1c563" path="/var/lib/kubelet/pods/b35e59e6-59f9-43bb-8dba-293ac3a1c563/volumes" Oct 30 03:08:36.116428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.116407 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.116532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.116436 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.116532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.116458 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.116632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.116609 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.116678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.116647 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf5w8\" (UniqueName: \"kubernetes.io/projected/f3405637-4232-4d64-b5cd-03b9508a7e41-kube-api-access-kf5w8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.116678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.116674 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.116809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.116791 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3405637-4232-4d64-b5cd-03b9508a7e41-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.217332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.217301 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf5w8\" (UniqueName: \"kubernetes.io/projected/f3405637-4232-4d64-b5cd-03b9508a7e41-kube-api-access-kf5w8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.217480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.217338 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.217480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.217359 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3405637-4232-4d64-b5cd-03b9508a7e41-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.217480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.217388 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.217480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.217406 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.217480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.217425 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.217480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.217468 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.218237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.218179 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.220049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.220013 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.220131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.220070 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.220229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.220212 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3405637-4232-4d64-b5cd-03b9508a7e41-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.220275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.220259 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f3405637-4232-4d64-b5cd-03b9508a7e41-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.235632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.235565 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf5w8\" (UniqueName: \"kubernetes.io/projected/f3405637-4232-4d64-b5cd-03b9508a7e41-kube-api-access-kf5w8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.235792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.235774 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3405637-4232-4d64-b5cd-03b9508a7e41-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tcgcs\" (UID: \"f3405637-4232-4d64-b5cd-03b9508a7e41\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.288458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.288434 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:36.595203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.595181 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs"] Oct 30 03:08:36.597125 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:08:36.597101 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3405637_4232_4d64_b5cd_03b9508a7e41.slice/crio-cbaefaa28762a6ff3c7a77457f1a61ad15762b04d69e1d1ea4b341b3e97c81bd WatchSource:0}: Error finding container cbaefaa28762a6ff3c7a77457f1a61ad15762b04d69e1d1ea4b341b3e97c81bd: Status 404 returned error can't find the container with id cbaefaa28762a6ff3c7a77457f1a61ad15762b04d69e1d1ea4b341b3e97c81bd Oct 30 03:08:36.598986 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.598953 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:08:36.599063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.599017 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:08:36.937413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:36.937344 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:37.041597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.041557 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:37.137443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.137405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:37.245429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.245352 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:37.335195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.335163 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" event={"ID":"f3405637-4232-4d64-b5cd-03b9508a7e41","Type":"ContainerStarted","Data":"5974a325bab1d68e0fe8710d545c612ce1bd71fc0bda2af38e1bca2536ba49a1"} Oct 30 03:08:37.335352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.335210 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" event={"ID":"f3405637-4232-4d64-b5cd-03b9508a7e41","Type":"ContainerStarted","Data":"cbaefaa28762a6ff3c7a77457f1a61ad15762b04d69e1d1ea4b341b3e97c81bd"} Oct 30 03:08:37.335404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.335364 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:37.336674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.336647 4442 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tcgcs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:08:37.336824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.336700 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" podUID="f3405637-4232-4d64-b5cd-03b9508a7e41" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:08:37.347787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.347743 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:37.378822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.378740 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" podStartSLOduration=2.378724225 podStartE2EDuration="2.378724225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:37.378646353 +0000 UTC m=+341.380999572" watchObservedRunningTime="2026-10-30 03:08:37.378724225 +0000 UTC m=+341.381077424" Oct 30 03:08:37.448626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.448591 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:37.718907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:37.718879 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:38.339518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.339490 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tcgcs" Oct 30 03:08:38.421682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.421650 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf"] Oct 30 03:08:38.421888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.421862 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" podUID="33ebce7e-462f-4bc1-8d26-af92d31e5846" containerName="discovery" containerID="cri-o://de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c" gracePeriod=30 Oct 30 03:08:38.658835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.658812 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:08:38.738535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.738505 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-dns-cert\") pod \"33ebce7e-462f-4bc1-8d26-af92d31e5846\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " Oct 30 03:08:38.739021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.738554 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-kube-api-access-jcrhq\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-kube-api-access-jcrhq\") pod \"33ebce7e-462f-4bc1-8d26-af92d31e5846\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " Oct 30 03:08:38.739021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.738589 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ebce7e-462f-4bc1-8d26-af92d31e5846-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ebce7e-462f-4bc1-8d26-af92d31e5846-local-certs\") pod \"33ebce7e-462f-4bc1-8d26-af92d31e5846\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " Oct 30 03:08:38.739021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.738636 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-ca-configmap\") pod \"33ebce7e-462f-4bc1-8d26-af92d31e5846\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " Oct 30 03:08:38.739021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.738672 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-token\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-token\") pod \"33ebce7e-462f-4bc1-8d26-af92d31e5846\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " Oct 30 03:08:38.739021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.738700 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-kubeconfig\") pod \"33ebce7e-462f-4bc1-8d26-af92d31e5846\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " Oct 30 03:08:38.739021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.738723 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-cacerts\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-cacerts\") pod \"33ebce7e-462f-4bc1-8d26-af92d31e5846\" (UID: \"33ebce7e-462f-4bc1-8d26-af92d31e5846\") " Oct 30 03:08:38.739275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.739056 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-ca-configmap" pod "33ebce7e-462f-4bc1-8d26-af92d31e5846" (UID: "33ebce7e-462f-4bc1-8d26-af92d31e5846"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:38.741046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.741008 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-dns-cert" pod "33ebce7e-462f-4bc1-8d26-af92d31e5846" (UID: "33ebce7e-462f-4bc1-8d26-af92d31e5846"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:38.741345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.741317 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-token" pod "33ebce7e-462f-4bc1-8d26-af92d31e5846" (UID: "33ebce7e-462f-4bc1-8d26-af92d31e5846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:38.741468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.741451 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-kube-api-access-jcrhq" pod "33ebce7e-462f-4bc1-8d26-af92d31e5846" (UID: "33ebce7e-462f-4bc1-8d26-af92d31e5846"). InnerVolumeSpecName "kube-api-access-jcrhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:38.741555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.741540 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-kubeconfig" pod "33ebce7e-462f-4bc1-8d26-af92d31e5846" (UID: "33ebce7e-462f-4bc1-8d26-af92d31e5846"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:38.741595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.741549 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ebce7e-462f-4bc1-8d26-af92d31e5846-local-certs" pod "33ebce7e-462f-4bc1-8d26-af92d31e5846" (UID: "33ebce7e-462f-4bc1-8d26-af92d31e5846"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:08:38.741595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.741568 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-cacerts" pod "33ebce7e-462f-4bc1-8d26-af92d31e5846" (UID: "33ebce7e-462f-4bc1-8d26-af92d31e5846"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:38.840029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.840004 4442 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-dns-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:38.840029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.840025 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcrhq\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-kube-api-access-jcrhq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:38.840150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.840037 4442 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33ebce7e-462f-4bc1-8d26-af92d31e5846-local-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:38.840150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.840046 4442 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-csr-ca-configmap\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:38.840150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.840055 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:38.840150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.840064 4442 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-istio-kubeconfig\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:38.840150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:38.840072 4442 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33ebce7e-462f-4bc1-8d26-af92d31e5846-cacerts\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:08:39.343897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.343859 4442 generic.go:356] "Generic (PLEG): container finished" podID="33ebce7e-462f-4bc1-8d26-af92d31e5846" containerID="de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c" exitCode=0 Oct 30 03:08:39.344057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.343917 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" Oct 30 03:08:39.344057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.343946 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" event={"ID":"33ebce7e-462f-4bc1-8d26-af92d31e5846","Type":"ContainerDied","Data":"de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c"} Oct 30 03:08:39.344057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.343998 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf" event={"ID":"33ebce7e-462f-4bc1-8d26-af92d31e5846","Type":"ContainerDied","Data":"5683fd445462a21eca401d0f190a658842f5dca28a338bd80710cf13d9434b15"} Oct 30 03:08:39.344057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.344018 4442 scope.go:122] "RemoveContainer" containerID="de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c" Oct 30 03:08:39.352916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.352897 4442 scope.go:122] "RemoveContainer" containerID="de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c" Oct 30 03:08:39.353181 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:08:39.353165 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c\": container with ID starting with de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c not found: ID does not exist" containerID="de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c" Oct 30 03:08:39.353218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.353191 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c"} err="failed to get container status \"de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c\": rpc error: code = NotFound desc = could not find container \"de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c\": container with ID starting with de6c1377bd27ffb8280f502837dedf158ffe86165e0bd4dc9f9628149ea4ef6c not found: ID does not exist" Oct 30 03:08:39.390633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.390606 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf"] Oct 30 03:08:39.401655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.401635 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-l4skf"] Oct 30 03:08:39.830694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.830658 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:39.930849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:39.930819 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:40.030305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:40.030276 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:40.104944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:40.104883 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ebce7e-462f-4bc1-8d26-af92d31e5846" path="/var/lib/kubelet/pods/33ebce7e-462f-4bc1-8d26-af92d31e5846/volumes" Oct 30 03:08:40.131441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:40.131411 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:40.231285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:40.231251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:40.331188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:40.331163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:42.648180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:42.648143 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:46.557422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:46.557388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:46.668560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:46.668526 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:51.095099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:51.095063 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:51.202859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:51.202823 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:52.555078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.555038 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:52.667825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.667795 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:52.758887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.758857 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d"] Oct 30 03:08:52.759147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.759134 4442 cpu_manager.go:414] "containerMap: removing container" podUID="33ebce7e-462f-4bc1-8d26-af92d31e5846" containerName="discovery" Oct 30 03:08:52.759185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.759151 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ebce7e-462f-4bc1-8d26-af92d31e5846" containerName="discovery" Oct 30 03:08:52.759251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.759238 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ebce7e-462f-4bc1-8d26-af92d31e5846" containerName="discovery" Oct 30 03:08:52.765120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.765099 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.779623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.779599 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 03:08:52.779827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.779809 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 03:08:52.790865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.790843 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 03:08:52.790957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.790883 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 03:08:52.790957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.790888 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8gxpp\"" Oct 30 03:08:52.794229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.792870 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d"] Oct 30 03:08:52.848142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.848061 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqvk4\" (UniqueName: \"kubernetes.io/projected/a9143c70-8cde-4c6b-942a-19747fb52a96-kube-api-access-gqvk4\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.848142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.848113 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-maas-api-tls\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.848142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.848132 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9143c70-8cde-4c6b-942a-19747fb52a96-openshift-service-ca\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.848378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.848230 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-metrics-certs\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.890352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.890321 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:08:52.949782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.949730 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqvk4\" (UniqueName: \"kubernetes.io/projected/a9143c70-8cde-4c6b-942a-19747fb52a96-kube-api-access-gqvk4\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.949915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.949832 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-maas-api-tls\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.949915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.949864 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9143c70-8cde-4c6b-942a-19747fb52a96-openshift-service-ca\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.949915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.949891 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-metrics-certs\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.950611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.950556 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9143c70-8cde-4c6b-942a-19747fb52a96-openshift-service-ca\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.952552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.952528 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-metrics-certs\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.952624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.952575 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-maas-api-tls\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:52.968580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:52.968548 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqvk4\" (UniqueName: \"kubernetes.io/projected/a9143c70-8cde-4c6b-942a-19747fb52a96-kube-api-access-gqvk4\") pod \"maas-api-8c7f85f5-w6q5d\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:53.078132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:53.078091 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:53.383343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:53.383320 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d"] Oct 30 03:08:53.385300 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:08:53.385272 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9143c70_8cde_4c6b_942a_19747fb52a96.slice/crio-ae449070b45de10de17827c89f05a2320fbe0d84348eac2aade1cb7b640e1c84 WatchSource:0}: Error finding container ae449070b45de10de17827c89f05a2320fbe0d84348eac2aade1cb7b640e1c84: Status 404 returned error can't find the container with id ae449070b45de10de17827c89f05a2320fbe0d84348eac2aade1cb7b640e1c84 Oct 30 03:08:53.397795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:53.397770 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" event={"ID":"a9143c70-8cde-4c6b-942a-19747fb52a96","Type":"ContainerStarted","Data":"ae449070b45de10de17827c89f05a2320fbe0d84348eac2aade1cb7b640e1c84"} Oct 30 03:08:54.644200 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:54.643484 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:08:54.862961 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:08:54.862922 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:08:55.412455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:55.412366 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" event={"ID":"a9143c70-8cde-4c6b-942a-19747fb52a96","Type":"ContainerStarted","Data":"cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8"} Oct 30 03:08:55.412616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:55.412523 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:08:55.451352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:08:55.451300 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" podStartSLOduration=1.808022637 podStartE2EDuration="3.45128768s" totalImagesPullingTime="1.643265043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:55.450218513 +0000 UTC m=+359.452571713" watchObservedRunningTime="2026-10-30 03:08:55.45128768 +0000 UTC m=+359.453640877" Oct 30 03:09:01.421711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:01.421682 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:09:01.855932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:01.855898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:09:02.005523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.005484 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:09:02.424233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.424199 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js"] Oct 30 03:09:02.429104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.429088 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.456995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.456965 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js"] Oct 30 03:09:02.533384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.533343 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-maas-api-tls\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.533542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.533412 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-metrics-certs\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.533542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.533510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ddc6c-c347-4ba8-b6ee-6e80ba285321-openshift-service-ca\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.533625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.533548 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txnvg\" (UniqueName: \"kubernetes.io/projected/987ddc6c-c347-4ba8-b6ee-6e80ba285321-kube-api-access-txnvg\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.634193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.634158 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ddc6c-c347-4ba8-b6ee-6e80ba285321-openshift-service-ca\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.634385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.634200 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txnvg\" (UniqueName: \"kubernetes.io/projected/987ddc6c-c347-4ba8-b6ee-6e80ba285321-kube-api-access-txnvg\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.634385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.634242 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-maas-api-tls\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.634385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.634284 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-metrics-certs\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.635065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.634984 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ddc6c-c347-4ba8-b6ee-6e80ba285321-openshift-service-ca\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.636969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.636939 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-metrics-certs\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.637364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.637322 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-maas-api-tls\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.653630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.653605 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txnvg\" (UniqueName: \"kubernetes.io/projected/987ddc6c-c347-4ba8-b6ee-6e80ba285321-kube-api-access-txnvg\") pod \"maas-api-65b844bfb4-vk8js\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:02.746622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:02.746567 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:03.043731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:03.043696 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js"] Oct 30 03:09:03.046541 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:09:03.046509 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987ddc6c_c347_4ba8_b6ee_6e80ba285321.slice/crio-6add5f56c894142eaa51220ec3a2c03959ac240a14afb6c7f614dda22842a4aa WatchSource:0}: Error finding container 6add5f56c894142eaa51220ec3a2c03959ac240a14afb6c7f614dda22842a4aa: Status 404 returned error can't find the container with id 6add5f56c894142eaa51220ec3a2c03959ac240a14afb6c7f614dda22842a4aa Oct 30 03:09:03.448665 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:03.448628 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" event={"ID":"987ddc6c-c347-4ba8-b6ee-6e80ba285321","Type":"ContainerStarted","Data":"683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282"} Oct 30 03:09:03.449156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:03.448695 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" event={"ID":"987ddc6c-c347-4ba8-b6ee-6e80ba285321","Type":"ContainerStarted","Data":"6add5f56c894142eaa51220ec3a2c03959ac240a14afb6c7f614dda22842a4aa"} Oct 30 03:09:03.449156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:03.448744 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:03.488739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:03.488687 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" podStartSLOduration=1.201678663 podStartE2EDuration="1.488674557s" totalImagesPullingTime="286.995894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:03.488256763 +0000 UTC m=+367.490609961" watchObservedRunningTime="2026-10-30 03:09:03.488674557 +0000 UTC m=+367.491027754" Oct 30 03:09:09.457903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.457875 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 03:09:09.524906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.524872 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d"] Oct 30 03:09:09.525190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.525162 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" podUID="a9143c70-8cde-4c6b-942a-19747fb52a96" containerName="maas-api" containerID="cri-o://cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8" gracePeriod=30 Oct 30 03:09:09.763916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.763892 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:09:09.850654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.850616 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:09:09.890505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.890479 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-metrics-certs\") pod \"a9143c70-8cde-4c6b-942a-19747fb52a96\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " Oct 30 03:09:09.890630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.890509 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-maas-api-tls\") pod \"a9143c70-8cde-4c6b-942a-19747fb52a96\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " Oct 30 03:09:09.890630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.890566 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9143c70-8cde-4c6b-942a-19747fb52a96-kube-api-access-gqvk4\" (UniqueName: \"kubernetes.io/projected/a9143c70-8cde-4c6b-942a-19747fb52a96-kube-api-access-gqvk4\") pod \"a9143c70-8cde-4c6b-942a-19747fb52a96\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " Oct 30 03:09:09.890630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.890600 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9143c70-8cde-4c6b-942a-19747fb52a96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9143c70-8cde-4c6b-942a-19747fb52a96-openshift-service-ca\") pod \"a9143c70-8cde-4c6b-942a-19747fb52a96\" (UID: \"a9143c70-8cde-4c6b-942a-19747fb52a96\") " Oct 30 03:09:09.891002 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.890979 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9143c70-8cde-4c6b-942a-19747fb52a96-openshift-service-ca" pod "a9143c70-8cde-4c6b-942a-19747fb52a96" (UID: "a9143c70-8cde-4c6b-942a-19747fb52a96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:09.892959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.892936 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-metrics-certs" pod "a9143c70-8cde-4c6b-942a-19747fb52a96" (UID: "a9143c70-8cde-4c6b-942a-19747fb52a96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:09.893055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.892965 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-maas-api-tls" pod "a9143c70-8cde-4c6b-942a-19747fb52a96" (UID: "a9143c70-8cde-4c6b-942a-19747fb52a96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:09.893055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.892978 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9143c70-8cde-4c6b-942a-19747fb52a96-kube-api-access-gqvk4" pod "a9143c70-8cde-4c6b-942a-19747fb52a96" (UID: "a9143c70-8cde-4c6b-942a-19747fb52a96"). InnerVolumeSpecName "kube-api-access-gqvk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:09.983083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.983052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:09:09.991087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.991064 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqvk4\" (UniqueName: \"kubernetes.io/projected/a9143c70-8cde-4c6b-942a-19747fb52a96-kube-api-access-gqvk4\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:09:09.991087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.991087 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9143c70-8cde-4c6b-942a-19747fb52a96-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:09:09.991194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.991096 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:09:09.991194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:09.991105 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9143c70-8cde-4c6b-942a-19747fb52a96-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:09:10.477872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.477794 4442 generic.go:356] "Generic (PLEG): container finished" podID="a9143c70-8cde-4c6b-942a-19747fb52a96" containerID="cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8" exitCode=0 Oct 30 03:09:10.477872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.477841 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" event={"ID":"a9143c70-8cde-4c6b-942a-19747fb52a96","Type":"ContainerDied","Data":"cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8"} Oct 30 03:09:10.477872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.477849 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" Oct 30 03:09:10.478379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.477882 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d" event={"ID":"a9143c70-8cde-4c6b-942a-19747fb52a96","Type":"ContainerDied","Data":"ae449070b45de10de17827c89f05a2320fbe0d84348eac2aade1cb7b640e1c84"} Oct 30 03:09:10.478379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.477905 4442 scope.go:122] "RemoveContainer" containerID="cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8" Oct 30 03:09:10.485954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.485937 4442 scope.go:122] "RemoveContainer" containerID="cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8" Oct 30 03:09:10.486222 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:09:10.486204 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8\": container with ID starting with cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8 not found: ID does not exist" containerID="cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8" Oct 30 03:09:10.486269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.486232 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8"} err="failed to get container status \"cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8\": rpc error: code = NotFound desc = could not find container \"cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8\": container with ID starting with cafc696111b7a56be529ba54fa1391a0d46a572a7103e41d97cbd7b8ff6877e8 not found: ID does not exist" Oct 30 03:09:10.523497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.523475 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d"] Oct 30 03:09:10.526550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:10.526531 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8c7f85f5-w6q5d"] Oct 30 03:09:12.104675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:12.104641 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9143c70-8cde-4c6b-942a-19747fb52a96" path="/var/lib/kubelet/pods/a9143c70-8cde-4c6b-942a-19747fb52a96/volumes" Oct 30 03:09:15.658925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:15.658888 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:09:24.859046 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:09:24.859009 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:09:42.750432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:09:42.750347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:09:54.859677 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:09:54.859633 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:10:19.295863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.295829 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-kzdr9"] Oct 30 03:10:19.296356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.296100 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a9143c70-8cde-4c6b-942a-19747fb52a96" containerName="maas-api" Oct 30 03:10:19.296356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.296110 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9143c70-8cde-4c6b-942a-19747fb52a96" containerName="maas-api" Oct 30 03:10:19.296356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.296169 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9143c70-8cde-4c6b-942a-19747fb52a96" containerName="maas-api" Oct 30 03:10:19.297150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.297122 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:19.299060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.299046 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.318136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.318114 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 03:10:19.323602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.323576 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-trsxr\"" Oct 30 03:10:19.323698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.323613 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 03:10:19.324694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.324676 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-kzdr9"] Oct 30 03:10:19.385768 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.385728 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98d9l\" (UniqueName: \"kubernetes.io/projected/1b43267b-c9e7-4265-976a-6623a59284db-kube-api-access-98d9l\") pod \"ai-gateway-controller-7d85675f98-kzdr9\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.385882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.385779 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b43267b-c9e7-4265-976a-6623a59284db-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-kzdr9\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.398087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.398060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:19.486219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.486195 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98d9l\" (UniqueName: \"kubernetes.io/projected/1b43267b-c9e7-4265-976a-6623a59284db-kube-api-access-98d9l\") pod \"ai-gateway-controller-7d85675f98-kzdr9\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.486329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.486233 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b43267b-c9e7-4265-976a-6623a59284db-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-kzdr9\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.488863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.488838 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b43267b-c9e7-4265-976a-6623a59284db-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-kzdr9\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.504816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.504792 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98d9l\" (UniqueName: \"kubernetes.io/projected/1b43267b-c9e7-4265-976a-6623a59284db-kube-api-access-98d9l\") pod \"ai-gateway-controller-7d85675f98-kzdr9\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.611788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.611719 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:19.903877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:19.903855 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-kzdr9"] Oct 30 03:10:19.906059 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:10:19.906036 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b43267b_c9e7_4265_976a_6623a59284db.slice/crio-7b7ea50377a9c1beabb613bfc1428c18b5b471bea2175180fc52e46a672c84d5 WatchSource:0}: Error finding container 7b7ea50377a9c1beabb613bfc1428c18b5b471bea2175180fc52e46a672c84d5: Status 404 returned error can't find the container with id 7b7ea50377a9c1beabb613bfc1428c18b5b471bea2175180fc52e46a672c84d5 Oct 30 03:10:20.755004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:20.754967 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" event={"ID":"1b43267b-c9e7-4265-976a-6623a59284db","Type":"ContainerStarted","Data":"7b7ea50377a9c1beabb613bfc1428c18b5b471bea2175180fc52e46a672c84d5"} Oct 30 03:10:21.760026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:21.759991 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" event={"ID":"1b43267b-c9e7-4265-976a-6623a59284db","Type":"ContainerStarted","Data":"7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3"} Oct 30 03:10:21.760406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:21.760064 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:21.797857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:21.797809 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" podStartSLOduration=1.314313469 podStartE2EDuration="2.797796167s" totalImagesPullingTime="1.483482698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:21.796493036 +0000 UTC m=+445.798846256" watchObservedRunningTime="2026-10-30 03:10:21.797796167 +0000 UTC m=+445.800149389" Oct 30 03:10:23.559950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.559913 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:23.662609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.662561 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:23.772610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.772583 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v"] Oct 30 03:10:23.775953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.775937 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:23.806577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.806552 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 03:10:23.806779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.806637 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vsjpp\"" Oct 30 03:10:23.807922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.807895 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v"] Oct 30 03:10:23.819212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.819154 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca299bea-7668-4ecc-80e9-73dcefad570d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n9x2v\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:23.819310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.819292 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hgj\" (UniqueName: \"kubernetes.io/projected/ca299bea-7668-4ecc-80e9-73dcefad570d-kube-api-access-m8hgj\") pod \"payload-pre-processing-7b98bb678c-n9x2v\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:23.854878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.854854 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hgcd9"] Oct 30 03:10:23.857944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.857929 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:23.893275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.893254 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hgcd9"] Oct 30 03:10:23.920169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.920149 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hgj\" (UniqueName: \"kubernetes.io/projected/ca299bea-7668-4ecc-80e9-73dcefad570d-kube-api-access-m8hgj\") pod \"payload-pre-processing-7b98bb678c-n9x2v\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:23.920277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.920186 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmdc\" (UniqueName: \"kubernetes.io/projected/9f652e0b-e10a-47c6-971b-080c5b78b53a-kube-api-access-skmdc\") pod \"payload-processing-5f78687d5b-hgcd9\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:23.920277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.920213 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f652e0b-e10a-47c6-971b-080c5b78b53a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hgcd9\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:23.920277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.920236 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca299bea-7668-4ecc-80e9-73dcefad570d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n9x2v\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:23.921145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.921125 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca299bea-7668-4ecc-80e9-73dcefad570d-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-n9x2v\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:23.940046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:23.940026 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hgj\" (UniqueName: \"kubernetes.io/projected/ca299bea-7668-4ecc-80e9-73dcefad570d-kube-api-access-m8hgj\") pod \"payload-pre-processing-7b98bb678c-n9x2v\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:24.020853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.020814 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmdc\" (UniqueName: \"kubernetes.io/projected/9f652e0b-e10a-47c6-971b-080c5b78b53a-kube-api-access-skmdc\") pod \"payload-processing-5f78687d5b-hgcd9\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:24.021025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.020875 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f652e0b-e10a-47c6-971b-080c5b78b53a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hgcd9\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:24.021564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.021540 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f652e0b-e10a-47c6-971b-080c5b78b53a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-hgcd9\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:24.039077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.039046 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmdc\" (UniqueName: \"kubernetes.io/projected/9f652e0b-e10a-47c6-971b-080c5b78b53a-kube-api-access-skmdc\") pod \"payload-processing-5f78687d5b-hgcd9\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:24.088186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.088112 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:24.170567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.170532 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:24.382498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.382471 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v"] Oct 30 03:10:24.384660 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:10:24.384635 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca299bea_7668_4ecc_80e9_73dcefad570d.slice/crio-1b86f95b12ccced3f1cff23333d016827754f81f00d832a4568c6e9a11e2ff7a WatchSource:0}: Error finding container 1b86f95b12ccced3f1cff23333d016827754f81f00d832a4568c6e9a11e2ff7a: Status 404 returned error can't find the container with id 1b86f95b12ccced3f1cff23333d016827754f81f00d832a4568c6e9a11e2ff7a Oct 30 03:10:24.463973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.463856 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hgcd9"] Oct 30 03:10:24.466018 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:10:24.465987 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f652e0b_e10a_47c6_971b_080c5b78b53a.slice/crio-cae6d5289a35c456dd0af09d2737bf7192c735b7f09d902ed9903ecc4b0b16cb WatchSource:0}: Error finding container cae6d5289a35c456dd0af09d2737bf7192c735b7f09d902ed9903ecc4b0b16cb: Status 404 returned error can't find the container with id cae6d5289a35c456dd0af09d2737bf7192c735b7f09d902ed9903ecc4b0b16cb Oct 30 03:10:24.774385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.774346 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" event={"ID":"9f652e0b-e10a-47c6-971b-080c5b78b53a","Type":"ContainerStarted","Data":"cae6d5289a35c456dd0af09d2737bf7192c735b7f09d902ed9903ecc4b0b16cb"} Oct 30 03:10:24.775539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:24.775517 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" event={"ID":"ca299bea-7668-4ecc-80e9-73dcefad570d","Type":"ContainerStarted","Data":"1b86f95b12ccced3f1cff23333d016827754f81f00d832a4568c6e9a11e2ff7a"} Oct 30 03:10:24.861093 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:10:24.861049 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:10:27.791664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:27.791629 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" event={"ID":"9f652e0b-e10a-47c6-971b-080c5b78b53a","Type":"ContainerStarted","Data":"69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b"} Oct 30 03:10:27.792178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:27.791774 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:27.793363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:27.793340 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" event={"ID":"ca299bea-7668-4ecc-80e9-73dcefad570d","Type":"ContainerStarted","Data":"f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817"} Oct 30 03:10:27.793478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:27.793387 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:27.827767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:27.827695 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" podStartSLOduration=2.51176106 podStartE2EDuration="4.827678613s" totalImagesPullingTime="2.315917553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:27.826582519 +0000 UTC m=+451.828935698" watchObservedRunningTime="2026-10-30 03:10:27.827678613 +0000 UTC m=+451.830031811" Oct 30 03:10:27.863502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:27.863458 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" podStartSLOduration=2.499026056 podStartE2EDuration="4.863446253s" totalImagesPullingTime="2.364420197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:27.8624819 +0000 UTC m=+451.864835097" watchObservedRunningTime="2026-10-30 03:10:27.863446253 +0000 UTC m=+451.865799449" Oct 30 03:10:32.766854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:32.766813 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:33.023573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.023506 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-b5fdc8966-ztpcp"] Oct 30 03:10:33.027141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.027121 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.046732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.046702 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b5fdc8966-ztpcp"] Oct 30 03:10:33.102471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.102438 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz84w\" (UniqueName: \"kubernetes.io/projected/09e1682b-def2-4c93-8bd9-0f90131c27d9-kube-api-access-dz84w\") pod \"ai-gateway-controller-b5fdc8966-ztpcp\" (UID: \"09e1682b-def2-4c93-8bd9-0f90131c27d9\") " pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.102601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.102481 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09e1682b-def2-4c93-8bd9-0f90131c27d9-metrics-certs\") pod \"ai-gateway-controller-b5fdc8966-ztpcp\" (UID: \"09e1682b-def2-4c93-8bd9-0f90131c27d9\") " pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.203748 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.203714 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09e1682b-def2-4c93-8bd9-0f90131c27d9-metrics-certs\") pod \"ai-gateway-controller-b5fdc8966-ztpcp\" (UID: \"09e1682b-def2-4c93-8bd9-0f90131c27d9\") " pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.203902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.203868 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz84w\" (UniqueName: \"kubernetes.io/projected/09e1682b-def2-4c93-8bd9-0f90131c27d9-kube-api-access-dz84w\") pod \"ai-gateway-controller-b5fdc8966-ztpcp\" (UID: \"09e1682b-def2-4c93-8bd9-0f90131c27d9\") " pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.206208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.206185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09e1682b-def2-4c93-8bd9-0f90131c27d9-metrics-certs\") pod \"ai-gateway-controller-b5fdc8966-ztpcp\" (UID: \"09e1682b-def2-4c93-8bd9-0f90131c27d9\") " pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.221970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.221946 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz84w\" (UniqueName: \"kubernetes.io/projected/09e1682b-def2-4c93-8bd9-0f90131c27d9-kube-api-access-dz84w\") pod \"ai-gateway-controller-b5fdc8966-ztpcp\" (UID: \"09e1682b-def2-4c93-8bd9-0f90131c27d9\") " pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.338805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.338707 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:33.630604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.630463 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b5fdc8966-ztpcp"] Oct 30 03:10:33.632884 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:10:33.632858 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e1682b_def2_4c93_8bd9_0f90131c27d9.slice/crio-b8bb2a04e40e1a74ce9f07f72f63a4c6afab19a307e9bac9a16c0f044b778346 WatchSource:0}: Error finding container b8bb2a04e40e1a74ce9f07f72f63a4c6afab19a307e9bac9a16c0f044b778346: Status 404 returned error can't find the container with id b8bb2a04e40e1a74ce9f07f72f63a4c6afab19a307e9bac9a16c0f044b778346 Oct 30 03:10:33.817789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:33.817744 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" event={"ID":"09e1682b-def2-4c93-8bd9-0f90131c27d9","Type":"ContainerStarted","Data":"b8bb2a04e40e1a74ce9f07f72f63a4c6afab19a307e9bac9a16c0f044b778346"} Oct 30 03:10:34.823586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:34.823553 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" event={"ID":"09e1682b-def2-4c93-8bd9-0f90131c27d9","Type":"ContainerStarted","Data":"254a446cb673ea393c484ad0dd28eef1e83e986f4424e7e620becdb7d05946ea"} Oct 30 03:10:34.824010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:34.823647 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:34.860015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:34.859974 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" podStartSLOduration=1.117283432 podStartE2EDuration="1.859958909s" totalImagesPullingTime="742.675477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:34.859074429 +0000 UTC m=+458.861427626" watchObservedRunningTime="2026-10-30 03:10:34.859958909 +0000 UTC m=+458.862312105" Oct 30 03:10:38.799990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:38.799964 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 03:10:38.800425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:38.800025 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 03:10:42.873984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:42.873950 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:45.830432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:45.830399 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-b5fdc8966-ztpcp" Oct 30 03:10:45.890683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:45.890652 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-kzdr9"] Oct 30 03:10:45.890944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:45.890888 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" podUID="1b43267b-c9e7-4265-976a-6623a59284db" containerName="manager" containerID="cri-o://7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3" gracePeriod=30 Oct 30 03:10:46.130911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.130889 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:46.200681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.200650 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b43267b-c9e7-4265-976a-6623a59284db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b43267b-c9e7-4265-976a-6623a59284db-metrics-certs\") pod \"1b43267b-c9e7-4265-976a-6623a59284db\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " Oct 30 03:10:46.200845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.200725 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b43267b-c9e7-4265-976a-6623a59284db-kube-api-access-98d9l\" (UniqueName: \"kubernetes.io/projected/1b43267b-c9e7-4265-976a-6623a59284db-kube-api-access-98d9l\") pod \"1b43267b-c9e7-4265-976a-6623a59284db\" (UID: \"1b43267b-c9e7-4265-976a-6623a59284db\") " Oct 30 03:10:46.202810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.202778 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b43267b-c9e7-4265-976a-6623a59284db-metrics-certs" pod "1b43267b-c9e7-4265-976a-6623a59284db" (UID: "1b43267b-c9e7-4265-976a-6623a59284db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:10:46.202957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.202934 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b43267b-c9e7-4265-976a-6623a59284db-kube-api-access-98d9l" pod "1b43267b-c9e7-4265-976a-6623a59284db" (UID: "1b43267b-c9e7-4265-976a-6623a59284db"). InnerVolumeSpecName "kube-api-access-98d9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:10:46.301919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.301885 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b43267b-c9e7-4265-976a-6623a59284db-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:10:46.301919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.301917 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98d9l\" (UniqueName: \"kubernetes.io/projected/1b43267b-c9e7-4265-976a-6623a59284db-kube-api-access-98d9l\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:10:46.871764 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.871730 4442 generic.go:356] "Generic (PLEG): container finished" podID="1b43267b-c9e7-4265-976a-6623a59284db" containerID="7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3" exitCode=0 Oct 30 03:10:46.872183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.871793 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" Oct 30 03:10:46.872183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.871819 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" event={"ID":"1b43267b-c9e7-4265-976a-6623a59284db","Type":"ContainerDied","Data":"7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3"} Oct 30 03:10:46.872183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.871873 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-kzdr9" event={"ID":"1b43267b-c9e7-4265-976a-6623a59284db","Type":"ContainerDied","Data":"7b7ea50377a9c1beabb613bfc1428c18b5b471bea2175180fc52e46a672c84d5"} Oct 30 03:10:46.872183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.871895 4442 scope.go:122] "RemoveContainer" containerID="7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3" Oct 30 03:10:46.880388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.880372 4442 scope.go:122] "RemoveContainer" containerID="7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3" Oct 30 03:10:46.880631 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:10:46.880616 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3\": container with ID starting with 7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3 not found: ID does not exist" containerID="7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3" Oct 30 03:10:46.880676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.880639 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3"} err="failed to get container status \"7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3\": rpc error: code = NotFound desc = could not find container \"7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3\": container with ID starting with 7e7ef6119b1db0aa7b2becac52cb4c48a696fd85b3d9858da3702e12f91ec6d3 not found: ID does not exist" Oct 30 03:10:46.917381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.917352 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-kzdr9"] Oct 30 03:10:46.928509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:46.928487 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-kzdr9"] Oct 30 03:10:47.585031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:47.584992 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:48.104360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:48.104328 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b43267b-c9e7-4265-976a-6623a59284db" path="/var/lib/kubelet/pods/1b43267b-c9e7-4265-976a-6623a59284db/volumes" Oct 30 03:10:48.107340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:48.107315 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:49.414648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:49.414606 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:49.512447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:49.512405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:49.886657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:49.886625 4442 generic.go:356] "Generic (PLEG): container finished" podID="05381e85-4985-4f92-9d08-6cca60bf8059" containerID="3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911" exitCode=1 Oct 30 03:10:49.886830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:49.886701 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4dk5t" event={"ID":"05381e85-4985-4f92-9d08-6cca60bf8059","Type":"ContainerDied","Data":"3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911"} Oct 30 03:10:49.887052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:49.887037 4442 scope.go:122] "RemoveContainer" containerID="3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911" Oct 30 03:10:50.893431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:50.893393 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4dk5t" event={"ID":"05381e85-4985-4f92-9d08-6cca60bf8059","Type":"ContainerStarted","Data":"98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94"} Oct 30 03:10:50.893925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:50.893602 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:10:52.866648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.866613 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k"] Oct 30 03:10:52.867045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.866935 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1b43267b-c9e7-4265-976a-6623a59284db" containerName="manager" Oct 30 03:10:52.867045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.866946 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b43267b-c9e7-4265-976a-6623a59284db" containerName="manager" Oct 30 03:10:52.867045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.867010 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b43267b-c9e7-4265-976a-6623a59284db" containerName="manager" Oct 30 03:10:52.869616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.869596 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:52.894221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.894185 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 03:10:52.894340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.894251 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 03:10:52.894340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.894291 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9mjxw\"" Oct 30 03:10:52.894340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.894338 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 03:10:52.897215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.897190 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k"] Oct 30 03:10:52.942131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.942107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:52.954193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.954166 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:52.954276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.954208 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:52.954357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.954339 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:52.954415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.954373 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssxj\" (UniqueName: \"kubernetes.io/projected/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-kube-api-access-4ssxj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:52.954458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.954432 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:52.954502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:52.954462 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.055613 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.055584 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.055613 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.055619 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssxj\" (UniqueName: \"kubernetes.io/projected/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-kube-api-access-4ssxj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.055875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.055635 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.055875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.055653 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.055875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.055723 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.055875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.055798 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.056064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.056036 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.056124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.056085 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.056224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.056205 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.058090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.058068 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.058509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.058492 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.073947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.073926 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssxj\" (UniqueName: \"kubernetes.io/projected/cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d-kube-api-access-4ssxj\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k\" (UID: \"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.180356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.180276 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:53.472360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.472259 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k"] Oct 30 03:10:53.474389 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:10:53.474360 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdbebe0f_c13d_4fca_ba8a_b3efd2a6815d.slice/crio-7e74f82ae7020fe452e0a850884d6fb1a047596667a2a6c38e1fcdfb70cf2a73 WatchSource:0}: Error finding container 7e74f82ae7020fe452e0a850884d6fb1a047596667a2a6c38e1fcdfb70cf2a73: Status 404 returned error can't find the container with id 7e74f82ae7020fe452e0a850884d6fb1a047596667a2a6c38e1fcdfb70cf2a73 Oct 30 03:10:53.910593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:53.910561 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" event={"ID":"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d","Type":"ContainerStarted","Data":"7e74f82ae7020fe452e0a850884d6fb1a047596667a2a6c38e1fcdfb70cf2a73"} Oct 30 03:10:54.113118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:54.113083 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:10:54.859261 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:10:54.859226 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:10:54.915736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:54.915699 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" event={"ID":"cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d","Type":"ContainerStarted","Data":"a6cdbcb9fb75fa5b2e86ee829e48264ff97af0edd1d6a74a445fdcaf05011390"} Oct 30 03:10:54.953391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:54.953300 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" podStartSLOduration=1.755711431 podStartE2EDuration="2.953287432s" totalImagesPullingTime="1.197576001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:54.95295064 +0000 UTC m=+478.955303839" watchObservedRunningTime="2026-10-30 03:10:54.953287432 +0000 UTC m=+478.955640629" Oct 30 03:10:55.180839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:55.180809 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:10:56.567832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:56.567806 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:10:57.019919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:57.019883 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:10:58.594823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:10:58.594789 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:00.104943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.104913 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:11:00.677561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.677528 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4"] Oct 30 03:11:00.685465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.685439 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.701737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.701714 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4"] Oct 30 03:11:00.711056 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.711024 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 03:11:00.725995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.725970 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:00.826333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.826310 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c43243c-5d9f-46c9-89c6-816d34027ccc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.826465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.826339 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.826465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.826357 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzdp\" (UniqueName: \"kubernetes.io/projected/6c43243c-5d9f-46c9-89c6-816d34027ccc-kube-api-access-sdzdp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.826465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.826422 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.826567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.826478 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.826567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.826496 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927301 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c43243c-5d9f-46c9-89c6-816d34027ccc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927334 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927353 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzdp\" (UniqueName: \"kubernetes.io/projected/6c43243c-5d9f-46c9-89c6-816d34027ccc-kube-api-access-sdzdp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927407 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927452 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927485 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927793 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927870 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.927983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.927956 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.929862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.929839 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c43243c-5d9f-46c9-89c6-816d34027ccc-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.930154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.930137 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c43243c-5d9f-46c9-89c6-816d34027ccc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.946979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.946955 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzdp\" (UniqueName: \"kubernetes.io/projected/6c43243c-5d9f-46c9-89c6-816d34027ccc-kube-api-access-sdzdp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k7xc4\" (UID: \"6c43243c-5d9f-46c9-89c6-816d34027ccc\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:00.997849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:00.997820 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:01.291018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:01.290987 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4"] Oct 30 03:11:01.293120 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:11:01.293092 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c43243c_5d9f_46c9_89c6_816d34027ccc.slice/crio-4bfc0fc0e26901dcdf3a107178227a14a053d65d8953a7626b23e99d2ea08a58 WatchSource:0}: Error finding container 4bfc0fc0e26901dcdf3a107178227a14a053d65d8953a7626b23e99d2ea08a58: Status 404 returned error can't find the container with id 4bfc0fc0e26901dcdf3a107178227a14a053d65d8953a7626b23e99d2ea08a58 Oct 30 03:11:01.900384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:01.900306 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 03:11:01.945996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:01.945962 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" event={"ID":"6c43243c-5d9f-46c9-89c6-816d34027ccc","Type":"ContainerStarted","Data":"f0821a5bf2f4037b32df8eba4196c8ecfd098c0202b2d0b42ee349bb11f2dc84"} Oct 30 03:11:01.946151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:01.946018 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" event={"ID":"6c43243c-5d9f-46c9-89c6-816d34027ccc","Type":"ContainerStarted","Data":"4bfc0fc0e26901dcdf3a107178227a14a053d65d8953a7626b23e99d2ea08a58"} Oct 30 03:11:01.984586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:01.984528 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" podStartSLOduration=1.780461839 podStartE2EDuration="1.984510505s" totalImagesPullingTime="204.048666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:01.983729259 +0000 UTC m=+485.986082483" watchObservedRunningTime="2026-10-30 03:11:01.984510505 +0000 UTC m=+485.986863703" Oct 30 03:11:01.998528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:01.998502 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:02.122052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:02.122022 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:03.180399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.180366 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:11:03.193011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.192985 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:11:03.331116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.331084 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bcrdv"] Oct 30 03:11:03.334465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.334446 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.350485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.350464 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 03:11:03.355314 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.355290 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bcrdv"] Oct 30 03:11:03.397353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.397329 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:03.448163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.448078 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-home\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.448163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.448124 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvlxg\" (UniqueName: \"kubernetes.io/projected/1f31cbac-b475-488d-9098-f7e3e4a24071-kube-api-access-cvlxg\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.448163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.448153 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.448414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.448270 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.448414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.448304 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.448414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.448387 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f31cbac-b475-488d-9098-f7e3e4a24071-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.549485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.549457 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.549649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.549530 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.549649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.549556 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.549649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.549615 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f31cbac-b475-488d-9098-f7e3e4a24071-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.549843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.549674 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-home\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.549843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.549697 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvlxg\" (UniqueName: \"kubernetes.io/projected/1f31cbac-b475-488d-9098-f7e3e4a24071-kube-api-access-cvlxg\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.550073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.550040 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.550294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.550264 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.550544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.550499 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-home\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.552463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.552443 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f31cbac-b475-488d-9098-f7e3e4a24071-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.552648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.552633 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f31cbac-b475-488d-9098-f7e3e4a24071-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.567918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.567900 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvlxg\" (UniqueName: \"kubernetes.io/projected/1f31cbac-b475-488d-9098-f7e3e4a24071-kube-api-access-cvlxg\") pod \"echo-simulator-kserve-64b6f8f96c-bcrdv\" (UID: \"1f31cbac-b475-488d-9098-f7e3e4a24071\") " pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.646816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.646792 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:03.943959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.943933 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-bcrdv"] Oct 30 03:11:03.945728 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:11:03.945696 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f31cbac_b475_488d_9098_f7e3e4a24071.slice/crio-56138713b07335f32cfedb4b2bfee7e0996f422d8d82a853bd477ecfd7fe1404 WatchSource:0}: Error finding container 56138713b07335f32cfedb4b2bfee7e0996f422d8d82a853bd477ecfd7fe1404: Status 404 returned error can't find the container with id 56138713b07335f32cfedb4b2bfee7e0996f422d8d82a853bd477ecfd7fe1404 Oct 30 03:11:03.955122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.955093 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" event={"ID":"1f31cbac-b475-488d-9098-f7e3e4a24071","Type":"ContainerStarted","Data":"56138713b07335f32cfedb4b2bfee7e0996f422d8d82a853bd477ecfd7fe1404"} Oct 30 03:11:03.966514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:03.966491 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k" Oct 30 03:11:04.380041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:04.380011 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:11:05.711529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:05.711438 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:05.966321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:05.966239 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" event={"ID":"1f31cbac-b475-488d-9098-f7e3e4a24071","Type":"ContainerStarted","Data":"79e0ec0f157a20271b83815e477c5012df4fcfb2a714957be7a454e084af26b8"} Oct 30 03:11:06.646985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:06.646958 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:07.048937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.048886 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" podStartSLOduration=2.586768814 podStartE2EDuration="4.04887132s" totalImagesPullingTime="1.462102506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:06.131142995 +0000 UTC m=+490.133496191" watchObservedRunningTime="2026-10-30 03:11:07.04887132 +0000 UTC m=+491.051224518" Oct 30 03:11:07.049592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.049571 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v"] Oct 30 03:11:07.053552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.053531 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.082197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.082175 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v"] Oct 30 03:11:07.100015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.099987 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:07.106492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.106469 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 03:11:07.181338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.181305 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc4sc\" (UniqueName: \"kubernetes.io/projected/b83c6124-3537-4816-b12a-ae756634bcc6-kube-api-access-bc4sc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.181497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.181361 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b83c6124-3537-4816-b12a-ae756634bcc6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.181497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.181453 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.181589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.181491 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.181589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.181534 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.181589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.181572 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.282503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.282435 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc4sc\" (UniqueName: \"kubernetes.io/projected/b83c6124-3537-4816-b12a-ae756634bcc6-kube-api-access-bc4sc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.282699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.282549 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b83c6124-3537-4816-b12a-ae756634bcc6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.282699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.282605 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.282699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.282662 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.282903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.282699 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.282903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.282738 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.283443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.283413 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.283565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.283443 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.283565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.283481 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.285992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.285966 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b83c6124-3537-4816-b12a-ae756634bcc6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.286241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.286219 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b83c6124-3537-4816-b12a-ae756634bcc6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.301298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.301243 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc4sc\" (UniqueName: \"kubernetes.io/projected/b83c6124-3537-4816-b12a-ae756634bcc6-kube-api-access-bc4sc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v\" (UID: \"b83c6124-3537-4816-b12a-ae756634bcc6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.376264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.376233 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:07.670553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.670529 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v"] Oct 30 03:11:07.672703 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:11:07.672674 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb83c6124_3537_4816_b12a_ae756634bcc6.slice/crio-8e959881524250ef2d086f728035d20a64bb1a29f1a29dc695cb1825662a948f WatchSource:0}: Error finding container 8e959881524250ef2d086f728035d20a64bb1a29f1a29dc695cb1825662a948f: Status 404 returned error can't find the container with id 8e959881524250ef2d086f728035d20a64bb1a29f1a29dc695cb1825662a948f Oct 30 03:11:07.977071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.977035 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" event={"ID":"b83c6124-3537-4816-b12a-ae756634bcc6","Type":"ContainerStarted","Data":"ad7884964715cf046261dd0f7256259d29d24656aeccc5956568566beb206a81"} Oct 30 03:11:07.977220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:07.977088 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" event={"ID":"b83c6124-3537-4816-b12a-ae756634bcc6","Type":"ContainerStarted","Data":"8e959881524250ef2d086f728035d20a64bb1a29f1a29dc695cb1825662a948f"} Oct 30 03:11:08.017279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:08.016009 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" podStartSLOduration=0.794181661 podStartE2EDuration="1.015992557s" totalImagesPullingTime="221.810896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:08.015371721 +0000 UTC m=+492.017724919" watchObservedRunningTime="2026-10-30 03:11:08.015992557 +0000 UTC m=+492.018345755" Oct 30 03:11:08.377302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:08.377273 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:10.998281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:10.998191 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:11.010819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:11.010795 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:12.004823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:12.004793 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k7xc4" Oct 30 03:11:13.647255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:13.647219 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:13.659777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:13.659741 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:14.013570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:14.013543 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-bcrdv" Oct 30 03:11:17.313863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.313828 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz"] Oct 30 03:11:17.316972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.316957 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.341033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.341001 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fqfn6\"" Oct 30 03:11:17.350786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.350764 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz"] Oct 30 03:11:17.358836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.358814 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.358945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.358849 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.358945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.358879 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.359080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.358952 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.359080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.359061 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.359177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.359104 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.359177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.359132 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hhf\" (UniqueName: \"kubernetes.io/projected/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-kube-api-access-j9hhf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.359240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.359205 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.359240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.359230 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.376972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.376951 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:17.390219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.390194 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:17.459825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.459788 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.459834 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.459879 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.459905 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.459941 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.459978 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460000 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hhf\" (UniqueName: \"kubernetes.io/projected/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-kube-api-access-j9hhf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460033 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460063 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460302 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460377 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460462 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460484 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.460933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.460874 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.462492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.462470 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.462574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.462513 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.479109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.479084 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.479385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.479368 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hhf\" (UniqueName: \"kubernetes.io/projected/d2d45fb8-f876-4f98-9f0f-72bb69aaaa68-kube-api-access-j9hhf\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-vtlvz\" (UID: \"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.630558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.630493 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:17.847905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.847873 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:17.937117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.937093 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz"] Oct 30 03:11:17.939133 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:11:17.939101 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d45fb8_f876_4f98_9f0f_72bb69aaaa68.slice/crio-c1b916620432f720dcc940d90d962fc2437769bed79fb8f58616c7120ecf392f WatchSource:0}: Error finding container c1b916620432f720dcc940d90d962fc2437769bed79fb8f58616c7120ecf392f: Status 404 returned error can't find the container with id c1b916620432f720dcc940d90d962fc2437769bed79fb8f58616c7120ecf392f Oct 30 03:11:17.955308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:17.955281 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:18.020876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:18.020842 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" event={"ID":"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68","Type":"ContainerStarted","Data":"c1b916620432f720dcc940d90d962fc2437769bed79fb8f58616c7120ecf392f"} Oct 30 03:11:18.031973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:18.031951 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v" Oct 30 03:11:18.054679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:18.054652 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:11:20.775713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:20.775684 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:20.776021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:20.775772 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:20.776021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:20.775805 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:21.037008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:21.036931 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" event={"ID":"d2d45fb8-f876-4f98-9f0f-72bb69aaaa68","Type":"ContainerStarted","Data":"c451b30d0bc3b4da0ab56c2dc3fe2e8ef5ae658e0a4221356ccbfd6a81eb3064"} Oct 30 03:11:21.101942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:21.101895 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podStartSLOduration=1.268115262 podStartE2EDuration="4.101872696s" totalImagesPullingTime="2.833757434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:21.090935658 +0000 UTC m=+505.093288869" watchObservedRunningTime="2026-10-30 03:11:21.101872696 +0000 UTC m=+505.104225889" Oct 30 03:11:21.631664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:21.631634 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:21.632867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:21.632839 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 03:11:21.632990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:21.632896 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 03:11:22.631861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:22.631823 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 03:11:22.632335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:22.631882 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 03:11:22.655073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:22.655042 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:22.762898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:22.762861 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:23.631378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:23.631343 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 03:11:23.631554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:23.631397 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 03:11:24.248402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:24.248368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:24.426292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:24.426251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:24.631441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:24.631407 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 03:11:24.631610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:24.631457 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 03:11:24.859196 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:11:24.859161 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:11:25.631414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:25.631374 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 03:11:25.631964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:25.631438 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 03:11:25.705187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:25.705153 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:26.631016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:26.630982 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 03:11:26.631193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:26.631051 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 03:11:27.631136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:27.631096 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 03:11:27.631629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:27.631157 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 03:11:28.923139 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:28.923105 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:28.923634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:28.923460 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:28.924434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:28.924415 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" Oct 30 03:11:32.298964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.298931 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw"] Oct 30 03:11:32.303313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.303291 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.329212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.329189 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw"] Oct 30 03:11:32.406201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406169 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406203 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406245 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406291 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pzz\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-kube-api-access-b4pzz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406351 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406375 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406458 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406499 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.406593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.406568 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.507799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.507740 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.507799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.507798 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pzz\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-kube-api-access-b4pzz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.507999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.507826 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.507999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.507854 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.507999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.507896 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.507999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.507923 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.507999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.507961 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.508247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.508031 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.508247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.508076 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.508350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.508300 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.508350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.508303 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.508429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.508380 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.508429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.508389 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.508720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.508700 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.510338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.510313 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.510580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.510564 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.526140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.526114 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.526248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.526175 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pzz\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-kube-api-access-b4pzz\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-qb9nw\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.615689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.615611 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:32.945797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.945772 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw"] Oct 30 03:11:32.947787 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:11:32.947746 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5871f77f_4012_4c6e_a0ef_ffb5814bcdc5.slice/crio-3f5027478ac1388bf7bc3c7c7bfb1dde516d92a021f08a2d822e3a26201488e1 WatchSource:0}: Error finding container 3f5027478ac1388bf7bc3c7c7bfb1dde516d92a021f08a2d822e3a26201488e1: Status 404 returned error can't find the container with id 3f5027478ac1388bf7bc3c7c7bfb1dde516d92a021f08a2d822e3a26201488e1 Oct 30 03:11:32.949901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.949866 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:32.949976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.949933 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:32.949976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:32.949964 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:33.088817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:33.088779 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" event={"ID":"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5","Type":"ContainerStarted","Data":"156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827"} Oct 30 03:11:33.088998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:33.088845 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" event={"ID":"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5","Type":"ContainerStarted","Data":"3f5027478ac1388bf7bc3c7c7bfb1dde516d92a021f08a2d822e3a26201488e1"} Oct 30 03:11:33.131022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:33.130968 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podStartSLOduration=1.130951326 podStartE2EDuration="1.130951326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:33.129650981 +0000 UTC m=+517.132004204" watchObservedRunningTime="2026-10-30 03:11:33.130951326 +0000 UTC m=+517.133304522" Oct 30 03:11:33.616328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:33.616291 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:33.617985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:33.617958 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 03:11:33.618104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:33.618007 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 03:11:34.616041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:34.616003 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 03:11:34.616195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:34.616065 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 03:11:35.616688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:35.616650 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 03:11:35.617128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:35.616720 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 03:11:36.616226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:36.616189 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 03:11:36.616406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:36.616241 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 03:11:37.616104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:37.616061 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 03:11:37.616535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:37.616134 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 03:11:38.616289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:38.616253 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 03:11:38.616722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:38.616307 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 03:11:39.616958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:39.616920 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 03:11:39.617354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:39.616999 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 03:11:41.064089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:41.064039 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:41.064546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:41.064331 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:41.065187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:41.065167 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:11:42.987875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:42.987839 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:54.859615 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:11:54.859577 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:11:55.657747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:55.657710 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:55.855726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:55.855694 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:55.970909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:55.970821 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:56.127202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.127143 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:56.460588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.460555 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:56.560302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.560266 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4"] Oct 30 03:11:56.567558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.567538 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.571001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.570979 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:56.587846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.587825 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c7c56c3d-openshift-default-dockercfg-tqfbx\"" Oct 30 03:11:56.601457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.601434 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4"] Oct 30 03:11:56.674035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.674010 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:56.720160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720099 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-certs\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720142 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-data\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720177 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-envoy\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720266 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efabf41c-6966-428b-a148-4841fecfbfa3-istio-podinfo\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720298 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-istio-token\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720323 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-socket\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efabf41c-6966-428b-a148-4841fecfbfa3-istiod-ca-cert\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720431 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-credential-socket\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.720464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.720449 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mwd\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-kube-api-access-m6mwd\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.780363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.780304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:56.820906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.820878 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-certs\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.820924 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-data\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821066 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-envoy\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821112 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efabf41c-6966-428b-a148-4841fecfbfa3-istio-podinfo\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821344 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-data\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821350 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-istio-token\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821385 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-certs\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821408 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-socket\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821433 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efabf41c-6966-428b-a148-4841fecfbfa3-istiod-ca-cert\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821477 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-credential-socket\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821501 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mwd\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-kube-api-access-m6mwd\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.821848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.821714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-socket\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.822209 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.822186 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-credential-socket\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.822276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.822241 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efabf41c-6966-428b-a148-4841fecfbfa3-istiod-ca-cert\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.823803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.823779 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efabf41c-6966-428b-a148-4841fecfbfa3-istio-podinfo\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.823911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.823894 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-envoy\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.840623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.840595 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mwd\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-kube-api-access-m6mwd\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.840889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.840866 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-istio-token\") pod \"e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.872783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.872745 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:56.880719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.880701 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:56.958911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:56.958876 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:57.193395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:57.193370 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4"] Oct 30 03:11:57.194718 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:11:57.194694 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefabf41c_6966_428b_a148_4841fecfbfa3.slice/crio-e1d7f6591b279aad54a150f75edaf5987b34468c7a0fad26f3b3a2666a1cce8c WatchSource:0}: Error finding container e1d7f6591b279aad54a150f75edaf5987b34468c7a0fad26f3b3a2666a1cce8c: Status 404 returned error can't find the container with id e1d7f6591b279aad54a150f75edaf5987b34468c7a0fad26f3b3a2666a1cce8c Oct 30 03:11:57.197003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:57.196965 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:57.197087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:57.197048 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:57.197128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:57.197090 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:11:57.205079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:57.205055 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" event={"ID":"efabf41c-6966-428b-a148-4841fecfbfa3","Type":"ContainerStarted","Data":"e1d7f6591b279aad54a150f75edaf5987b34468c7a0fad26f3b3a2666a1cce8c"} Oct 30 03:11:57.583467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:57.583432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:58.108299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:58.108264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:58.212158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:58.212113 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" event={"ID":"efabf41c-6966-428b-a148-4841fecfbfa3","Type":"ContainerStarted","Data":"db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859"} Oct 30 03:11:58.278120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:58.278065 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podStartSLOduration=2.278051203 podStartE2EDuration="2.278051203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:58.266564106 +0000 UTC m=+542.268917334" watchObservedRunningTime="2026-10-30 03:11:58.278051203 +0000 UTC m=+542.280404400" Oct 30 03:11:58.365388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:58.365310 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:58.881200 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:58.881167 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:58.886320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:58.886297 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:59.216652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:59.216565 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:59.217569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:59.217554 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:11:59.400028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:59.399996 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:59.527833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:59.527804 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:59.724307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:59.724279 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:11:59.831494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:11:59.831411 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:00.580489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.580448 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:00.729652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.729619 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42"] Oct 30 03:12:00.733655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.733635 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:00.753887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.753851 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-acace923\"" Oct 30 03:12:00.756726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.756697 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42"] Oct 30 03:12:00.778907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.778877 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:00.858651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.858559 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27pt\" (UniqueName: \"kubernetes.io/projected/8f94e947-d209-4059-8b0b-4dba3771cf65-kube-api-access-l27pt\") pod \"payload-processing-e2e-migrate-acace923-c764674-qvt42\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:00.858870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.858657 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94e947-d209-4059-8b0b-4dba3771cf65-plugins-config-volume\") pod \"payload-processing-e2e-migrate-acace923-c764674-qvt42\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:00.960138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.960106 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27pt\" (UniqueName: \"kubernetes.io/projected/8f94e947-d209-4059-8b0b-4dba3771cf65-kube-api-access-l27pt\") pod \"payload-processing-e2e-migrate-acace923-c764674-qvt42\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:00.960337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.960163 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94e947-d209-4059-8b0b-4dba3771cf65-plugins-config-volume\") pod \"payload-processing-e2e-migrate-acace923-c764674-qvt42\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:00.960924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.960905 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94e947-d209-4059-8b0b-4dba3771cf65-plugins-config-volume\") pod \"payload-processing-e2e-migrate-acace923-c764674-qvt42\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:00.978508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.978479 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27pt\" (UniqueName: \"kubernetes.io/projected/8f94e947-d209-4059-8b0b-4dba3771cf65-kube-api-access-l27pt\") pod \"payload-processing-e2e-migrate-acace923-c764674-qvt42\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:00.991684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.991655 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh"] Oct 30 03:12:00.995261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:00.995240 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.008881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.008859 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-acace923-serving-cert\"" Oct 30 03:12:01.018288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.018266 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh"] Oct 30 03:12:01.047395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.047366 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:01.061710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.061681 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-metrics-certs\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.061888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.061732 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-maas-api-tls\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.061942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.061926 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55qnk\" (UniqueName: \"kubernetes.io/projected/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-kube-api-access-55qnk\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.062033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.062018 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-openshift-service-ca\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.147644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.147611 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:01.162822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.162794 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-maas-api-tls\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.162969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.162859 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55qnk\" (UniqueName: \"kubernetes.io/projected/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-kube-api-access-55qnk\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.162969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.162906 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-openshift-service-ca\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.162969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.162953 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-metrics-certs\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.163556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.163533 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-openshift-service-ca\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.165436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.165417 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-maas-api-tls\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.165529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.165441 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-metrics-certs\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.180654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.180631 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55qnk\" (UniqueName: \"kubernetes.io/projected/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-kube-api-access-55qnk\") pod \"maas-api-e2e-migrate-acace923-5f65dcb696-f45dh\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.309583 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.309553 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:01.355347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.355324 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42"] Oct 30 03:12:01.357334 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:01.357296 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f94e947_d209_4059_8b0b_4dba3771cf65.slice/crio-25c76c70c659bca29108dbb7213e54375e0086a9bd1feaeb5c1f1dd1eb972fb0 WatchSource:0}: Error finding container 25c76c70c659bca29108dbb7213e54375e0086a9bd1feaeb5c1f1dd1eb972fb0: Status 404 returned error can't find the container with id 25c76c70c659bca29108dbb7213e54375e0086a9bd1feaeb5c1f1dd1eb972fb0 Oct 30 03:12:01.377001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.376975 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:01.615834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:01.615709 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh"] Oct 30 03:12:01.618091 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:01.618054 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e36caf4_e5a8_438d_a3cd_77439dd8fd6e.slice/crio-1d81d340c60093992e67e9ff112f5730804fcb21351a0a3873aec24096d38316 WatchSource:0}: Error finding container 1d81d340c60093992e67e9ff112f5730804fcb21351a0a3873aec24096d38316: Status 404 returned error can't find the container with id 1d81d340c60093992e67e9ff112f5730804fcb21351a0a3873aec24096d38316 Oct 30 03:12:02.242913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.242796 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" event={"ID":"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e","Type":"ContainerStarted","Data":"574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23"} Oct 30 03:12:02.242913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.242879 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:02.242913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.242904 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" event={"ID":"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e","Type":"ContainerStarted","Data":"1d81d340c60093992e67e9ff112f5730804fcb21351a0a3873aec24096d38316"} Oct 30 03:12:02.244310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.244282 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" event={"ID":"8f94e947-d209-4059-8b0b-4dba3771cf65","Type":"ContainerStarted","Data":"07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3"} Oct 30 03:12:02.244427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.244327 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" event={"ID":"8f94e947-d209-4059-8b0b-4dba3771cf65","Type":"ContainerStarted","Data":"25c76c70c659bca29108dbb7213e54375e0086a9bd1feaeb5c1f1dd1eb972fb0"} Oct 30 03:12:02.244427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.244359 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:02.286039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.285988 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" podStartSLOduration=1.91816189 podStartE2EDuration="2.285971764s" totalImagesPullingTime="367.809874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:02.28347296 +0000 UTC m=+546.285826160" watchObservedRunningTime="2026-10-30 03:12:02.285971764 +0000 UTC m=+546.288324952" Oct 30 03:12:02.323949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.323873 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" podStartSLOduration=2.323854851 podStartE2EDuration="2.323854851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:02.322543087 +0000 UTC m=+546.324896284" watchObservedRunningTime="2026-10-30 03:12:02.323854851 +0000 UTC m=+546.326208048" Oct 30 03:12:02.382221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.382186 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:02.515122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.515088 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2"] Oct 30 03:12:02.518733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.518710 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.528168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.528144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:02.532799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.532782 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a788fe69\"" Oct 30 03:12:02.536083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.536062 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a788fe69-dockercfg-9xgs6\"" Oct 30 03:12:02.537649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.537631 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2"] Oct 30 03:12:02.614909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.614870 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj"] Oct 30 03:12:02.619012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.618987 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:02.640394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.640316 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj"] Oct 30 03:12:02.675510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.675477 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6408d6-96fb-4167-afb8-8db79663cedc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.675684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.675517 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskpk\" (UniqueName: \"kubernetes.io/projected/8e6408d6-96fb-4167-afb8-8db79663cedc-kube-api-access-jskpk\") pod \"payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.689381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.689341 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:02.777038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.776903 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6408d6-96fb-4167-afb8-8db79663cedc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.777038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.776957 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d704ffc8-e35a-448e-85a4-f665f3d8de8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:02.777038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.776991 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjj29\" (UniqueName: \"kubernetes.io/projected/d704ffc8-e35a-448e-85a4-f665f3d8de8b-kube-api-access-kjj29\") pod \"payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:02.777038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.777027 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskpk\" (UniqueName: \"kubernetes.io/projected/8e6408d6-96fb-4167-afb8-8db79663cedc-kube-api-access-jskpk\") pod \"payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.777730 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.777703 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6408d6-96fb-4167-afb8-8db79663cedc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.800201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.800170 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskpk\" (UniqueName: \"kubernetes.io/projected/8e6408d6-96fb-4167-afb8-8db79663cedc-kube-api-access-jskpk\") pod \"payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.831561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.831529 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:02.845151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.845124 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:02.877671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.877636 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjj29\" (UniqueName: \"kubernetes.io/projected/d704ffc8-e35a-448e-85a4-f665f3d8de8b-kube-api-access-kjj29\") pod \"payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:02.877840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.877784 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d704ffc8-e35a-448e-85a4-f665f3d8de8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:02.878463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.878440 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d704ffc8-e35a-448e-85a4-f665f3d8de8b-plugins-config-volume\") pod \"payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:02.903786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.897305 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjj29\" (UniqueName: \"kubernetes.io/projected/d704ffc8-e35a-448e-85a4-f665f3d8de8b-kube-api-access-kjj29\") pod \"payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:02.948150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:02.948119 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:03.116956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.116884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:03.129646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.129626 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2"] Oct 30 03:12:03.132192 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:03.132157 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6408d6_96fb_4167_afb8_8db79663cedc.slice/crio-dfc03c0c47877a9742cb6e6da01c141bf58816e7c87f12a8c6cfb1c440981880 WatchSource:0}: Error finding container dfc03c0c47877a9742cb6e6da01c141bf58816e7c87f12a8c6cfb1c440981880: Status 404 returned error can't find the container with id dfc03c0c47877a9742cb6e6da01c141bf58816e7c87f12a8c6cfb1c440981880 Oct 30 03:12:03.250256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.250223 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" event={"ID":"8e6408d6-96fb-4167-afb8-8db79663cedc","Type":"ContainerStarted","Data":"71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75"} Oct 30 03:12:03.250463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.250281 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" event={"ID":"8e6408d6-96fb-4167-afb8-8db79663cedc","Type":"ContainerStarted","Data":"dfc03c0c47877a9742cb6e6da01c141bf58816e7c87f12a8c6cfb1c440981880"} Oct 30 03:12:03.250463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.250408 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:03.262638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.262612 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj"] Oct 30 03:12:03.264769 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:03.264729 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd704ffc8_e35a_448e_85a4_f665f3d8de8b.slice/crio-ea10dabd5d960477ea82625d04ba1d328abc0f3052f309d2bc76ba0c91fced85 WatchSource:0}: Error finding container ea10dabd5d960477ea82625d04ba1d328abc0f3052f309d2bc76ba0c91fced85: Status 404 returned error can't find the container with id ea10dabd5d960477ea82625d04ba1d328abc0f3052f309d2bc76ba0c91fced85 Oct 30 03:12:03.291239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.291174 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" podStartSLOduration=1.291155948 podStartE2EDuration="1.291155948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:03.289455292 +0000 UTC m=+547.291808489" watchObservedRunningTime="2026-10-30 03:12:03.291155948 +0000 UTC m=+547.293509222" Oct 30 03:12:03.777249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:03.777207 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:12:04.203315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.203247 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:04.256802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.256742 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" event={"ID":"d704ffc8-e35a-448e-85a4-f665f3d8de8b","Type":"ContainerStarted","Data":"fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202"} Oct 30 03:12:04.256966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.256825 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" event={"ID":"d704ffc8-e35a-448e-85a4-f665f3d8de8b","Type":"ContainerStarted","Data":"ea10dabd5d960477ea82625d04ba1d328abc0f3052f309d2bc76ba0c91fced85"} Oct 30 03:12:04.297510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.297452 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" podStartSLOduration=2.297438077 podStartE2EDuration="2.297438077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:04.295602829 +0000 UTC m=+548.297956028" watchObservedRunningTime="2026-10-30 03:12:04.297438077 +0000 UTC m=+548.299791273" Oct 30 03:12:04.330732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.330695 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:04.517304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.517274 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:04.701016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.700985 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx"] Oct 30 03:12:04.704739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.704715 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.717921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.717897 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-421ef828-serving-cert\"" Oct 30 03:12:04.737016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.736990 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx"] Oct 30 03:12:04.802177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.802100 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-openshift-service-ca\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.802177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.802135 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-metrics-certs\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.802616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.802295 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.802616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.802340 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6t2\" (UniqueName: \"kubernetes.io/projected/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-kube-api-access-sw6t2\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.805461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.805435 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:04.903634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.903601 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6t2\" (UniqueName: \"kubernetes.io/projected/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-kube-api-access-sw6t2\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.903820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.903652 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-openshift-service-ca\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.903820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.903670 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-metrics-certs\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.903820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.903731 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.904001 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:04.903855 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-serving-cert: secret "maas-api-e2e-ait-421ef828-serving-cert" not found Oct 30 03:12:04.904001 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:04.903916 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls podName:8c38b5a6-41c2-4d0f-955a-2c4a4b80e735 nodeName:}" failed. No retries permitted until 2026-10-30 03:12:05.403898576 +0000 UTC m=+549.406251768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls") pod "maas-api-e2e-ait-421ef828-5998d886bd-r99bx" (UID: "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735") : secret "maas-api-e2e-ait-421ef828-serving-cert" not found Oct 30 03:12:04.904414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.904390 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-openshift-service-ca\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.906426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.906401 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-metrics-certs\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.925110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.925078 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk"] Oct 30 03:12:04.925245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.925078 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6t2\" (UniqueName: \"kubernetes.io/projected/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-kube-api-access-sw6t2\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:04.928550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.928536 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:04.944852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.944828 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c7c56c3d\"" Oct 30 03:12:04.947037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.947015 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk"] Oct 30 03:12:04.955258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.955226 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:04.983185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:04.983158 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c7c56c3d-dockercfg-gb7rd\"" Oct 30 03:12:05.094454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.094377 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:05.106233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.106204 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qzz9\" (UniqueName: \"kubernetes.io/projected/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-kube-api-access-2qzz9\") pod \"payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:05.106334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.106310 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:05.207601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.207570 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qzz9\" (UniqueName: \"kubernetes.io/projected/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-kube-api-access-2qzz9\") pod \"payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:05.207785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.207628 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:05.208326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.208307 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:05.222423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.222379 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:05.225724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.225700 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qzz9\" (UniqueName: \"kubernetes.io/projected/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-kube-api-access-2qzz9\") pod \"payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:05.244532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.244505 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:05.262489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.262459 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:05.410111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.410026 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:05.412496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.412471 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls\") pod \"maas-api-e2e-ait-421ef828-5998d886bd-r99bx\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:05.544126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.544102 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk"] Oct 30 03:12:05.545930 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:05.545904 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46f7dad2_4fb4_46f1_8fd5_03ce90420da9.slice/crio-0fdbc8d3e68eaa98ccdf55e3e5c7d61a09f5e9d05c5e9b59188ae6b54edd1d13 WatchSource:0}: Error finding container 0fdbc8d3e68eaa98ccdf55e3e5c7d61a09f5e9d05c5e9b59188ae6b54edd1d13: Status 404 returned error can't find the container with id 0fdbc8d3e68eaa98ccdf55e3e5c7d61a09f5e9d05c5e9b59188ae6b54edd1d13 Oct 30 03:12:05.616690 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.616669 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:05.708835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.708794 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:05.834877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.834841 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:05.863591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.863561 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968"] Oct 30 03:12:05.868146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.868126 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:05.892039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.892016 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-421ef828\"" Oct 30 03:12:05.895298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.895259 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968"] Oct 30 03:12:05.935719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.935640 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx"] Oct 30 03:12:05.949971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:05.949950 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-421ef828-dockercfg-bqvwb\"" Oct 30 03:12:06.016268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.016237 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d28d52-d3be-4499-81c7-91d885472631-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:06.016268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.016273 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5sb5\" (UniqueName: \"kubernetes.io/projected/c1d28d52-d3be-4499-81c7-91d885472631-kube-api-access-z5sb5\") pod \"payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:06.117746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.117710 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d28d52-d3be-4499-81c7-91d885472631-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:06.117927 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.117746 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5sb5\" (UniqueName: \"kubernetes.io/projected/c1d28d52-d3be-4499-81c7-91d885472631-kube-api-access-z5sb5\") pod \"payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:06.118508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.118486 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d28d52-d3be-4499-81c7-91d885472631-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:06.135813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.135783 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5sb5\" (UniqueName: \"kubernetes.io/projected/c1d28d52-d3be-4499-81c7-91d885472631-kube-api-access-z5sb5\") pod \"payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:06.194846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.194782 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:06.274575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.274090 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" event={"ID":"46f7dad2-4fb4-46f1-8fd5-03ce90420da9","Type":"ContainerStarted","Data":"01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a"} Oct 30 03:12:06.274575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.274183 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" event={"ID":"46f7dad2-4fb4-46f1-8fd5-03ce90420da9","Type":"ContainerStarted","Data":"0fdbc8d3e68eaa98ccdf55e3e5c7d61a09f5e9d05c5e9b59188ae6b54edd1d13"} Oct 30 03:12:06.274575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.274231 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:06.276393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.275869 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" event={"ID":"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735","Type":"ContainerStarted","Data":"7666d091af1afd7b8933b212f61b44179d2f8fce0b14ade654c8c167d9e1c8c6"} Oct 30 03:12:06.312494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.312446 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" podStartSLOduration=2.312429592 podStartE2EDuration="2.312429592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:06.310372304 +0000 UTC m=+550.312725501" watchObservedRunningTime="2026-10-30 03:12:06.312429592 +0000 UTC m=+550.314782779" Oct 30 03:12:06.497061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.497039 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968"] Oct 30 03:12:06.499542 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:06.499512 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d28d52_d3be_4499_81c7_91d885472631.slice/crio-a81bd52b10d40f939aa2466296890537c6ff5435a2f9553bd3490b7c4ce1e006 WatchSource:0}: Error finding container a81bd52b10d40f939aa2466296890537c6ff5435a2f9553bd3490b7c4ce1e006: Status 404 returned error can't find the container with id a81bd52b10d40f939aa2466296890537c6ff5435a2f9553bd3490b7c4ce1e006 Oct 30 03:12:06.614233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.614199 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk"] Oct 30 03:12:06.617805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.617781 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.630732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.630712 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c7c56c3d-serving-cert\"" Oct 30 03:12:06.636459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.636440 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk"] Oct 30 03:12:06.716043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.715921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:06.724185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.724157 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhk9\" (UniqueName: \"kubernetes.io/projected/bab6add4-3b45-48a3-b8fa-3367d1879f1d-kube-api-access-dnhk9\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.724338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.724220 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-maas-api-tls\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.724338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.724332 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab6add4-3b45-48a3-b8fa-3367d1879f1d-openshift-service-ca\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.724455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.724353 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-metrics-certs\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.822871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.822828 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:06.824983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.824962 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab6add4-3b45-48a3-b8fa-3367d1879f1d-openshift-service-ca\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.825094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.824993 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-metrics-certs\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.825094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.825028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhk9\" (UniqueName: \"kubernetes.io/projected/bab6add4-3b45-48a3-b8fa-3367d1879f1d-kube-api-access-dnhk9\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.825094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.825087 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-maas-api-tls\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.825747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.825724 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab6add4-3b45-48a3-b8fa-3367d1879f1d-openshift-service-ca\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.827568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.827545 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-metrics-certs\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.827654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.827604 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-maas-api-tls\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.843633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.843611 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhk9\" (UniqueName: \"kubernetes.io/projected/bab6add4-3b45-48a3-b8fa-3367d1879f1d-kube-api-access-dnhk9\") pod \"maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:06.930312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:06.930277 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:07.247189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.247166 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk"] Oct 30 03:12:07.248986 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:07.248960 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab6add4_3b45_48a3_b8fa_3367d1879f1d.slice/crio-b9a0ec5c37714f0cf434b91cc2cb5ba0d7d644a9fa4774a3713bf2bbc2ec0558 WatchSource:0}: Error finding container b9a0ec5c37714f0cf434b91cc2cb5ba0d7d644a9fa4774a3713bf2bbc2ec0558: Status 404 returned error can't find the container with id b9a0ec5c37714f0cf434b91cc2cb5ba0d7d644a9fa4774a3713bf2bbc2ec0558 Oct 30 03:12:07.283006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.282977 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" event={"ID":"bab6add4-3b45-48a3-b8fa-3367d1879f1d","Type":"ContainerStarted","Data":"b9a0ec5c37714f0cf434b91cc2cb5ba0d7d644a9fa4774a3713bf2bbc2ec0558"} Oct 30 03:12:07.284689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.284661 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" event={"ID":"c1d28d52-d3be-4499-81c7-91d885472631","Type":"ContainerStarted","Data":"de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf"} Oct 30 03:12:07.284842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.284711 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" event={"ID":"c1d28d52-d3be-4499-81c7-91d885472631","Type":"ContainerStarted","Data":"a81bd52b10d40f939aa2466296890537c6ff5435a2f9553bd3490b7c4ce1e006"} Oct 30 03:12:07.284842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.284737 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:07.286496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.286476 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" event={"ID":"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735","Type":"ContainerStarted","Data":"e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836"} Oct 30 03:12:07.321213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.321158 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" podStartSLOduration=2.321142629 podStartE2EDuration="2.321142629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:07.31943732 +0000 UTC m=+551.321790518" watchObservedRunningTime="2026-10-30 03:12:07.321142629 +0000 UTC m=+551.323495828" Oct 30 03:12:07.356408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.356365 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" podStartSLOduration=2.895340304 podStartE2EDuration="3.356353525s" totalImagesPullingTime="461.013221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:07.355958273 +0000 UTC m=+551.358311479" watchObservedRunningTime="2026-10-30 03:12:07.356353525 +0000 UTC m=+551.358706721" Oct 30 03:12:07.415969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.415940 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:07.542571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:07.542503 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:08.255955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.255930 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:08.295357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.295311 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" event={"ID":"bab6add4-3b45-48a3-b8fa-3367d1879f1d","Type":"ContainerStarted","Data":"c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9"} Oct 30 03:12:08.295533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.295407 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:08.295533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.295520 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:08.340444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.340395 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" podStartSLOduration=2.035605727 podStartE2EDuration="2.340378984s" totalImagesPullingTime="304.773257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:08.339105551 +0000 UTC m=+552.341458760" watchObservedRunningTime="2026-10-30 03:12:08.340378984 +0000 UTC m=+552.342732182" Oct 30 03:12:08.509486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.509412 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm"] Oct 30 03:12:08.513275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.513252 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.526379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.526360 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a788fe69-serving-cert\"" Oct 30 03:12:08.534861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.534829 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm"] Oct 30 03:12:08.648916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.648880 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-maas-api-tls\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.649097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.649003 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-openshift-service-ca\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.649097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.649034 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtv25\" (UniqueName: \"kubernetes.io/projected/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-kube-api-access-rtv25\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.649185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.649139 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-metrics-certs\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.683934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.683901 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:08.750013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.749983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-maas-api-tls\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.750132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.750059 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-openshift-service-ca\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.750231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.750214 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtv25\" (UniqueName: \"kubernetes.io/projected/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-kube-api-access-rtv25\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.750299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.750282 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-metrics-certs\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.750726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.750701 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-openshift-service-ca\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.752765 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.752730 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-metrics-certs\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.752839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.752741 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-maas-api-tls\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.768013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.767966 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtv25\" (UniqueName: \"kubernetes.io/projected/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-kube-api-access-rtv25\") pod \"maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:08.799305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.799283 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:08.826346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:08.826313 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:09.140290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:09.140266 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm"] Oct 30 03:12:09.142606 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:09.142571 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8793b3a_498e_4ff6_b81f_ce3134ce32fd.slice/crio-ffe35e6039066380f26e8d838452e48acbfffc2635add1d0b29aabf3da0a8d48 WatchSource:0}: Error finding container ffe35e6039066380f26e8d838452e48acbfffc2635add1d0b29aabf3da0a8d48: Status 404 returned error can't find the container with id ffe35e6039066380f26e8d838452e48acbfffc2635add1d0b29aabf3da0a8d48 Oct 30 03:12:09.301535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:09.301495 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" event={"ID":"d8793b3a-498e-4ff6-b81f-ce3134ce32fd","Type":"ContainerStarted","Data":"ffe35e6039066380f26e8d838452e48acbfffc2635add1d0b29aabf3da0a8d48"} Oct 30 03:12:10.270460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:10.270425 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:10.308910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:10.308872 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" event={"ID":"d8793b3a-498e-4ff6-b81f-ce3134ce32fd","Type":"ContainerStarted","Data":"333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2"} Oct 30 03:12:10.309349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:10.308941 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:10.350124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:10.350067 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" podStartSLOduration=2.030118783 podStartE2EDuration="2.350051861s" totalImagesPullingTime="319.933078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:10.349460417 +0000 UTC m=+554.351813615" watchObservedRunningTime="2026-10-30 03:12:10.350051861 +0000 UTC m=+554.352405057" Oct 30 03:12:10.373980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:10.373935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:10.618444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:10.618363 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:12.197710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:12.197665 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:13.254063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:13.254037 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:13.934271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:13.934232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:13.978462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:13.978437 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:12:14.260546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:14.260516 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:14.307092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:14.307069 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:14.307777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:14.307741 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:15.712735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.712696 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj"] Oct 30 03:12:15.716448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.716427 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.729687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.729670 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3f9206f0-serving-cert\"" Oct 30 03:12:15.749188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.749148 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj"] Oct 30 03:12:15.821573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.821538 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-maas-api-tls\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.821804 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.821576 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfc8z\" (UniqueName: \"kubernetes.io/projected/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-kube-api-access-tfc8z\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.821897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.821860 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-openshift-service-ca\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.821959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.821922 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-metrics-certs\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.906777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.903643 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:15.922614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.922576 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-maas-api-tls\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.922614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.922615 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfc8z\" (UniqueName: \"kubernetes.io/projected/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-kube-api-access-tfc8z\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.922885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.922708 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-openshift-service-ca\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.922885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.922737 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-metrics-certs\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.923417 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.923390 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-openshift-service-ca\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.925239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.925210 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-maas-api-tls\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.925320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.925274 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-metrics-certs\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:15.941152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:15.941126 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfc8z\" (UniqueName: \"kubernetes.io/projected/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-kube-api-access-tfc8z\") pod \"maas-api-e2e-derive-3f9206f0-868544db57-xz9tj\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:16.030106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:16.030075 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:16.073923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:16.073860 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:16.279333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:16.279306 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:16.319676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:16.319608 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:16.354141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:16.353981 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj"] Oct 30 03:12:16.357546 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:16.357512 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e36b67_caac_4cd1_8a80_9a2c03a5e8dd.slice/crio-6489e5ac1521b324529150855bdad21a498d6efbfd85ae32cf02d747a0767526 WatchSource:0}: Error finding container 6489e5ac1521b324529150855bdad21a498d6efbfd85ae32cf02d747a0767526: Status 404 returned error can't find the container with id 6489e5ac1521b324529150855bdad21a498d6efbfd85ae32cf02d747a0767526 Oct 30 03:12:16.413780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:16.410062 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:17.309073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:17.309048 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:17.345667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:17.345633 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" event={"ID":"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd","Type":"ContainerStarted","Data":"51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624"} Oct 30 03:12:17.345832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:17.345692 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" event={"ID":"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd","Type":"ContainerStarted","Data":"6489e5ac1521b324529150855bdad21a498d6efbfd85ae32cf02d747a0767526"} Oct 30 03:12:17.345832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:17.345729 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:17.415964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:17.415911 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" podStartSLOduration=2.046112845 podStartE2EDuration="2.415896476s" totalImagesPullingTime="369.783631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:17.413169145 +0000 UTC m=+561.415522343" watchObservedRunningTime="2026-10-30 03:12:17.415896476 +0000 UTC m=+561.418249673" Oct 30 03:12:18.123588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:18.123556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:18.307577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:18.307548 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:20.225389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:20.225350 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:22.036158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:22.036125 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:23.357362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:23.357336 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:23.748934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:23.748895 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:24.387784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:24.387735 4442 generic.go:356] "Generic (PLEG): container finished" podID="efabf41c-6966-428b-a148-4841fecfbfa3" containerID="db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859" exitCode=0 Oct 30 03:12:24.388233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:24.387783 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" event={"ID":"efabf41c-6966-428b-a148-4841fecfbfa3","Type":"ContainerDied","Data":"db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859"} Oct 30 03:12:24.388233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:24.388166 4442 scope.go:122] "RemoveContainer" containerID="db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859" Oct 30 03:12:24.389416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:24.389389 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:24.389519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:24.389471 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:24.389587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:24.389518 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:24.859777 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:24.859706 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:12:25.419203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:25.419172 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" event={"ID":"efabf41c-6966-428b-a148-4841fecfbfa3","Type":"ContainerStarted","Data":"bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1"} Oct 30 03:12:25.759865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:25.759833 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:25.881401 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:25.881369 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:12:25.888778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:25.886858 4442 patch_prober.go:29] interesting pod/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 03:12:25.888778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:25.886914 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 03:12:26.223534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:26.223143 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:12:26.889780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:26.885875 4442 patch_prober.go:29] interesting pod/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 03:12:26.889780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:26.885939 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 03:12:27.592134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:27.592099 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:27.756389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:27.756355 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968"] Oct 30 03:12:27.876398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:27.876287 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:27.881331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:27.881303 4442 patch_prober.go:29] interesting pod/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 03:12:27.881446 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:27.881352 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 03:12:28.008779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:28.006527 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:28.881315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:28.881281 4442 patch_prober.go:29] interesting pod/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 03:12:28.881490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:28.881340 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 03:12:29.217062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:29.216986 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:12:29.902778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:29.891871 4442 patch_prober.go:29] interesting pod/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 03:12:29.902778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:29.891949 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 03:12:30.203239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:30.203149 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:30.886721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:30.886692 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:12:30.887626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:30.887609 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:12:32.185772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:32.185731 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:32.757203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:32.757170 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" podUID="c1d28d52-d3be-4499-81c7-91d885472631" containerName="payload-pre-processing" containerID="cri-o://de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf" gracePeriod=25 Oct 30 03:12:33.012293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.012235 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:33.099892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.099865 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d28d52-d3be-4499-81c7-91d885472631-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d28d52-d3be-4499-81c7-91d885472631-plugins-config-volume\") pod \"c1d28d52-d3be-4499-81c7-91d885472631\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " Oct 30 03:12:33.100045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.099927 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d28d52-d3be-4499-81c7-91d885472631-kube-api-access-z5sb5\" (UniqueName: \"kubernetes.io/projected/c1d28d52-d3be-4499-81c7-91d885472631-kube-api-access-z5sb5\") pod \"c1d28d52-d3be-4499-81c7-91d885472631\" (UID: \"c1d28d52-d3be-4499-81c7-91d885472631\") " Oct 30 03:12:33.100319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.100292 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d28d52-d3be-4499-81c7-91d885472631-plugins-config-volume" pod "c1d28d52-d3be-4499-81c7-91d885472631" (UID: "c1d28d52-d3be-4499-81c7-91d885472631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:33.102138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.102120 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d28d52-d3be-4499-81c7-91d885472631-kube-api-access-z5sb5" pod "c1d28d52-d3be-4499-81c7-91d885472631" (UID: "c1d28d52-d3be-4499-81c7-91d885472631"). InnerVolumeSpecName "kube-api-access-z5sb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:33.200673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.200646 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1d28d52-d3be-4499-81c7-91d885472631-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:33.200673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.200670 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5sb5\" (UniqueName: \"kubernetes.io/projected/c1d28d52-d3be-4499-81c7-91d885472631-kube-api-access-z5sb5\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:33.462463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.462379 4442 generic.go:356] "Generic (PLEG): container finished" podID="c1d28d52-d3be-4499-81c7-91d885472631" containerID="de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf" exitCode=0 Oct 30 03:12:33.462463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.462450 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" Oct 30 03:12:33.462463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.462453 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" event={"ID":"c1d28d52-d3be-4499-81c7-91d885472631","Type":"ContainerDied","Data":"de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf"} Oct 30 03:12:33.462713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.462506 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968" event={"ID":"c1d28d52-d3be-4499-81c7-91d885472631","Type":"ContainerDied","Data":"a81bd52b10d40f939aa2466296890537c6ff5435a2f9553bd3490b7c4ce1e006"} Oct 30 03:12:33.462713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.462527 4442 scope.go:122] "RemoveContainer" containerID="de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf" Oct 30 03:12:33.471845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.471745 4442 scope.go:122] "RemoveContainer" containerID="de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf" Oct 30 03:12:33.472175 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:33.472151 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf\": container with ID starting with de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf not found: ID does not exist" containerID="de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf" Oct 30 03:12:33.472271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.472184 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf"} err="failed to get container status \"de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf\": rpc error: code = NotFound desc = could not find container \"de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf\": container with ID starting with de3b3e3a1044612dc5dab428031ac19a72feb07455f04db1e96d8e53ae85beaf not found: ID does not exist" Oct 30 03:12:33.511381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.511352 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968"] Oct 30 03:12:33.522218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:33.522190 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-421ef828-7bd56c55f-jc968"] Oct 30 03:12:34.003266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:34.003233 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:34.106013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:34.105983 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d28d52-d3be-4499-81c7-91d885472631" path="/var/lib/kubelet/pods/c1d28d52-d3be-4499-81c7-91d885472631/volumes" Oct 30 03:12:34.765706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:34.765670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:35.011634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:35.011602 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42"] Oct 30 03:12:35.142390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:35.142295 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:35.241906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:35.241853 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:35.800429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:35.800388 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj"] Oct 30 03:12:35.831597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:35.831567 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2"] Oct 30 03:12:35.846261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:35.846234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:35.946741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:35.946712 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:36.047315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:36.047282 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:36.641112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:36.641083 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk"] Oct 30 03:12:36.750333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:36.750303 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:36.856449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:36.856415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:37.429513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.429473 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx"] Oct 30 03:12:37.429766 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.429701 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" podUID="8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" containerName="maas-api" containerID="cri-o://e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836" gracePeriod=30 Oct 30 03:12:37.481059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.481023 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:37.693688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.692827 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:37.745351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.745322 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-kube-api-access-sw6t2\" (UniqueName: \"kubernetes.io/projected/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-kube-api-access-sw6t2\") pod \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " Oct 30 03:12:37.745531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.745388 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls\") pod \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " Oct 30 03:12:37.745531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.745452 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-openshift-service-ca\") pod \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " Oct 30 03:12:37.745652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.745569 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-metrics-certs\") pod \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\" (UID: \"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735\") " Oct 30 03:12:37.745916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.745880 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-openshift-service-ca" pod "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" (UID: "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:37.747643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.747612 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls" pod "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" (UID: "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:37.747786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.747664 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-metrics-certs" pod "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" (UID: "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:37.747786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.747685 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-kube-api-access-sw6t2" pod "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" (UID: "8c38b5a6-41c2-4d0f-955a-2c4a4b80e735"). InnerVolumeSpecName "kube-api-access-sw6t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:37.852076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.852033 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:37.852512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.852115 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw6t2\" (UniqueName: \"kubernetes.io/projected/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-kube-api-access-sw6t2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:37.852512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.852134 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:37.852512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:37.852156 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:38.017925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.017896 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:38.118019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.117992 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:38.219541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.219510 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:38.489341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.489250 4442 generic.go:356] "Generic (PLEG): container finished" podID="8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" containerID="e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836" exitCode=0 Oct 30 03:12:38.489341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.489299 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" Oct 30 03:12:38.489341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.489319 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" event={"ID":"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735","Type":"ContainerDied","Data":"e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836"} Oct 30 03:12:38.489617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.489371 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx" event={"ID":"8c38b5a6-41c2-4d0f-955a-2c4a4b80e735","Type":"ContainerDied","Data":"7666d091af1afd7b8933b212f61b44179d2f8fce0b14ade654c8c167d9e1c8c6"} Oct 30 03:12:38.489617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.489395 4442 scope.go:122] "RemoveContainer" containerID="e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836" Oct 30 03:12:38.498521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.498498 4442 scope.go:122] "RemoveContainer" containerID="e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836" Oct 30 03:12:38.498783 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:38.498764 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836\": container with ID starting with e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836 not found: ID does not exist" containerID="e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836" Oct 30 03:12:38.498842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.498791 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836"} err="failed to get container status \"e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836\": rpc error: code = NotFound desc = could not find container \"e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836\": container with ID starting with e2b31e1a03ab88c8aa53c4b1499698a72e21777a6613317091c02d6fe0891836 not found: ID does not exist" Oct 30 03:12:38.536157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.536135 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx"] Oct 30 03:12:38.547046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.547019 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-421ef828-5998d886bd-r99bx"] Oct 30 03:12:38.833268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:38.833234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:39.662383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:39.662349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:39.727142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:39.727106 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk"] Oct 30 03:12:39.727493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:39.727326 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" podUID="bab6add4-3b45-48a3-b8fa-3367d1879f1d" containerName="maas-api" containerID="cri-o://c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9" gracePeriod=30 Oct 30 03:12:39.788604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:39.788565 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:39.989608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:39.989581 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:40.012952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.012904 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" podUID="8f94e947-d209-4059-8b0b-4dba3771cf65" containerName="payload-processing" containerID="cri-o://07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3" gracePeriod=25 Oct 30 03:12:40.073657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.073627 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab6add4-3b45-48a3-b8fa-3367d1879f1d-kube-api-access-dnhk9\" (UniqueName: \"kubernetes.io/projected/bab6add4-3b45-48a3-b8fa-3367d1879f1d-kube-api-access-dnhk9\") pod \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " Oct 30 03:12:40.073831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.073771 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-maas-api-tls\") pod \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " Oct 30 03:12:40.073831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.073823 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab6add4-3b45-48a3-b8fa-3367d1879f1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab6add4-3b45-48a3-b8fa-3367d1879f1d-openshift-service-ca\") pod \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " Oct 30 03:12:40.073976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.073874 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-metrics-certs\") pod \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\" (UID: \"bab6add4-3b45-48a3-b8fa-3367d1879f1d\") " Oct 30 03:12:40.074205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.074176 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab6add4-3b45-48a3-b8fa-3367d1879f1d-openshift-service-ca" pod "bab6add4-3b45-48a3-b8fa-3367d1879f1d" (UID: "bab6add4-3b45-48a3-b8fa-3367d1879f1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:40.075932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.075905 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab6add4-3b45-48a3-b8fa-3367d1879f1d-kube-api-access-dnhk9" pod "bab6add4-3b45-48a3-b8fa-3367d1879f1d" (UID: "bab6add4-3b45-48a3-b8fa-3367d1879f1d"). InnerVolumeSpecName "kube-api-access-dnhk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:40.076033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.075999 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-maas-api-tls" pod "bab6add4-3b45-48a3-b8fa-3367d1879f1d" (UID: "bab6add4-3b45-48a3-b8fa-3367d1879f1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:40.076033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.076007 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-metrics-certs" pod "bab6add4-3b45-48a3-b8fa-3367d1879f1d" (UID: "bab6add4-3b45-48a3-b8fa-3367d1879f1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:40.108273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.108238 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" path="/var/lib/kubelet/pods/8c38b5a6-41c2-4d0f-955a-2c4a4b80e735/volumes" Oct 30 03:12:40.159134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.159103 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:40.169800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.166382 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm"] Oct 30 03:12:40.169800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.166665 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" podUID="d8793b3a-498e-4ff6-b81f-ce3134ce32fd" containerName="maas-api" containerID="cri-o://333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2" gracePeriod=30 Oct 30 03:12:40.176148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.174739 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.176148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.174783 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bab6add4-3b45-48a3-b8fa-3367d1879f1d-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.176148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.174799 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bab6add4-3b45-48a3-b8fa-3367d1879f1d-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.176148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.174812 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnhk9\" (UniqueName: \"kubernetes.io/projected/bab6add4-3b45-48a3-b8fa-3367d1879f1d-kube-api-access-dnhk9\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.281561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.281413 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:40.306019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.305989 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:40.376091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.376063 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f94e947-d209-4059-8b0b-4dba3771cf65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94e947-d209-4059-8b0b-4dba3771cf65-plugins-config-volume\") pod \"8f94e947-d209-4059-8b0b-4dba3771cf65\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " Oct 30 03:12:40.376259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.376101 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f94e947-d209-4059-8b0b-4dba3771cf65-kube-api-access-l27pt\" (UniqueName: \"kubernetes.io/projected/8f94e947-d209-4059-8b0b-4dba3771cf65-kube-api-access-l27pt\") pod \"8f94e947-d209-4059-8b0b-4dba3771cf65\" (UID: \"8f94e947-d209-4059-8b0b-4dba3771cf65\") " Oct 30 03:12:40.376984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.376955 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f94e947-d209-4059-8b0b-4dba3771cf65-plugins-config-volume" pod "8f94e947-d209-4059-8b0b-4dba3771cf65" (UID: "8f94e947-d209-4059-8b0b-4dba3771cf65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:40.379376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.379351 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f94e947-d209-4059-8b0b-4dba3771cf65-kube-api-access-l27pt" pod "8f94e947-d209-4059-8b0b-4dba3771cf65" (UID: "8f94e947-d209-4059-8b0b-4dba3771cf65"). InnerVolumeSpecName "kube-api-access-l27pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:40.403964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.403943 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:40.420769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.420732 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:40.477533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.477504 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-kube-api-access-rtv25\" (UniqueName: \"kubernetes.io/projected/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-kube-api-access-rtv25\") pod \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " Oct 30 03:12:40.477685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.477576 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-metrics-certs\") pod \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " Oct 30 03:12:40.477685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.477656 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-openshift-service-ca\") pod \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " Oct 30 03:12:40.477829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.477686 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-maas-api-tls\") pod \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\" (UID: \"d8793b3a-498e-4ff6-b81f-ce3134ce32fd\") " Oct 30 03:12:40.477938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.477922 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f94e947-d209-4059-8b0b-4dba3771cf65-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.478002 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.477941 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l27pt\" (UniqueName: \"kubernetes.io/projected/8f94e947-d209-4059-8b0b-4dba3771cf65-kube-api-access-l27pt\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.478110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.478087 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-openshift-service-ca" pod "d8793b3a-498e-4ff6-b81f-ce3134ce32fd" (UID: "d8793b3a-498e-4ff6-b81f-ce3134ce32fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:40.479941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.479913 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-kube-api-access-rtv25" pod "d8793b3a-498e-4ff6-b81f-ce3134ce32fd" (UID: "d8793b3a-498e-4ff6-b81f-ce3134ce32fd"). InnerVolumeSpecName "kube-api-access-rtv25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:40.479941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.479922 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-metrics-certs" pod "d8793b3a-498e-4ff6-b81f-ce3134ce32fd" (UID: "d8793b3a-498e-4ff6-b81f-ce3134ce32fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:40.480067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.480004 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-maas-api-tls" pod "d8793b3a-498e-4ff6-b81f-ce3134ce32fd" (UID: "d8793b3a-498e-4ff6-b81f-ce3134ce32fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:40.500840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.500815 4442 generic.go:356] "Generic (PLEG): container finished" podID="8f94e947-d209-4059-8b0b-4dba3771cf65" containerID="07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3" exitCode=0 Oct 30 03:12:40.500946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.500855 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" Oct 30 03:12:40.500946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.500893 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" event={"ID":"8f94e947-d209-4059-8b0b-4dba3771cf65","Type":"ContainerDied","Data":"07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3"} Oct 30 03:12:40.501054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.500981 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42" event={"ID":"8f94e947-d209-4059-8b0b-4dba3771cf65","Type":"ContainerDied","Data":"25c76c70c659bca29108dbb7213e54375e0086a9bd1feaeb5c1f1dd1eb972fb0"} Oct 30 03:12:40.501054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.501013 4442 scope.go:122] "RemoveContainer" containerID="07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3" Oct 30 03:12:40.502226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.502202 4442 generic.go:356] "Generic (PLEG): container finished" podID="bab6add4-3b45-48a3-b8fa-3367d1879f1d" containerID="c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9" exitCode=0 Oct 30 03:12:40.502321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.502276 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" Oct 30 03:12:40.502391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.502273 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" event={"ID":"bab6add4-3b45-48a3-b8fa-3367d1879f1d","Type":"ContainerDied","Data":"c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9"} Oct 30 03:12:40.502445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.502406 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk" event={"ID":"bab6add4-3b45-48a3-b8fa-3367d1879f1d","Type":"ContainerDied","Data":"b9a0ec5c37714f0cf434b91cc2cb5ba0d7d644a9fa4774a3713bf2bbc2ec0558"} Oct 30 03:12:40.504039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.503942 4442 generic.go:356] "Generic (PLEG): container finished" podID="d8793b3a-498e-4ff6-b81f-ce3134ce32fd" containerID="333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2" exitCode=0 Oct 30 03:12:40.504039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.503979 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" event={"ID":"d8793b3a-498e-4ff6-b81f-ce3134ce32fd","Type":"ContainerDied","Data":"333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2"} Oct 30 03:12:40.504039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.504016 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" event={"ID":"d8793b3a-498e-4ff6-b81f-ce3134ce32fd","Type":"ContainerDied","Data":"ffe35e6039066380f26e8d838452e48acbfffc2635add1d0b29aabf3da0a8d48"} Oct 30 03:12:40.505528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.504049 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm" Oct 30 03:12:40.511732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.511712 4442 scope.go:122] "RemoveContainer" containerID="07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3" Oct 30 03:12:40.512047 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:40.512030 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3\": container with ID starting with 07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3 not found: ID does not exist" containerID="07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3" Oct 30 03:12:40.512090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.512053 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3"} err="failed to get container status \"07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3\": rpc error: code = NotFound desc = could not find container \"07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3\": container with ID starting with 07f806cc079aafc0b5c8ff3e01d5b952531967e6b6145bcbd0e6bbbe347edbf3 not found: ID does not exist" Oct 30 03:12:40.512090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.512068 4442 scope.go:122] "RemoveContainer" containerID="c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9" Oct 30 03:12:40.521293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.521276 4442 scope.go:122] "RemoveContainer" containerID="c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9" Oct 30 03:12:40.521576 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:40.521552 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9\": container with ID starting with c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9 not found: ID does not exist" containerID="c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9" Oct 30 03:12:40.521672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.521583 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9"} err="failed to get container status \"c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9\": rpc error: code = NotFound desc = could not find container \"c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9\": container with ID starting with c14b90cb36c1707594254c7d0c9808df4186be67b735ccb55108124d341649e9 not found: ID does not exist" Oct 30 03:12:40.521672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.521599 4442 scope.go:122] "RemoveContainer" containerID="333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2" Oct 30 03:12:40.521672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.521666 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:40.529554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.529537 4442 scope.go:122] "RemoveContainer" containerID="333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2" Oct 30 03:12:40.529876 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:40.529849 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2\": container with ID starting with 333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2 not found: ID does not exist" containerID="333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2" Oct 30 03:12:40.529935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.529882 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2"} err="failed to get container status \"333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2\": rpc error: code = NotFound desc = could not find container \"333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2\": container with ID starting with 333fea5ebd0821ee891649a506c1db0cd4442bae564653f02332b6fa66d295b2 not found: ID does not exist" Oct 30 03:12:40.562406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.562382 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk"] Oct 30 03:12:40.570021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.570003 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c7c56c3d-68bd44544b-wr5fk"] Oct 30 03:12:40.578598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.578580 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtv25\" (UniqueName: \"kubernetes.io/projected/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-kube-api-access-rtv25\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.578664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.578600 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.578664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.578610 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.578664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.578619 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8793b3a-498e-4ff6-b81f-ce3134ce32fd-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:40.604712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.604689 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42"] Oct 30 03:12:40.606640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.606621 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-acace923-c764674-qvt42"] Oct 30 03:12:40.647843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.647804 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm"] Oct 30 03:12:40.656660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.656630 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a788fe69-788dc88f48-gsqxm"] Oct 30 03:12:40.669145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.669097 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh"] Oct 30 03:12:40.669977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.669943 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" podUID="0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" containerName="maas-api" containerID="cri-o://574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23" gracePeriod=30 Oct 30 03:12:40.716949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.716910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:40.801667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.801617 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" podUID="d704ffc8-e35a-448e-85a4-f665f3d8de8b" containerName="payload-processing" containerID="cri-o://fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202" gracePeriod=25 Oct 30 03:12:40.832705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.832661 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" podUID="8e6408d6-96fb-4167-afb8-8db79663cedc" containerName="payload-pre-processing" containerID="cri-o://71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75" gracePeriod=25 Oct 30 03:12:40.856961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.856916 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:40.957274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.957245 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:40.973946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:40.973912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:41.076712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.076687 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:41.084058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.084035 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-metrics-certs\") pod \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " Oct 30 03:12:41.084216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.084201 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-kube-api-access-55qnk\" (UniqueName: \"kubernetes.io/projected/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-kube-api-access-55qnk\") pod \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " Oct 30 03:12:41.084276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.084252 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-openshift-service-ca\") pod \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " Oct 30 03:12:41.084335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.084282 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-maas-api-tls\") pod \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\" (UID: \"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e\") " Oct 30 03:12:41.084841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.084810 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-openshift-service-ca" pod "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" (UID: "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:41.086647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.086622 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-metrics-certs" pod "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" (UID: "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:41.087999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.087969 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-maas-api-tls" pod "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" (UID: "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:41.087999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.087984 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-kube-api-access-55qnk" pod "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" (UID: "0e36caf4-e5a8-438d-a3cd-77439dd8fd6e"). InnerVolumeSpecName "kube-api-access-55qnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:41.133321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.133292 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:41.185489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185458 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6408d6-96fb-4167-afb8-8db79663cedc-kube-api-access-jskpk\" (UniqueName: \"kubernetes.io/projected/8e6408d6-96fb-4167-afb8-8db79663cedc-kube-api-access-jskpk\") pod \"8e6408d6-96fb-4167-afb8-8db79663cedc\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " Oct 30 03:12:41.185647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185510 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d704ffc8-e35a-448e-85a4-f665f3d8de8b-kube-api-access-kjj29\" (UniqueName: \"kubernetes.io/projected/d704ffc8-e35a-448e-85a4-f665f3d8de8b-kube-api-access-kjj29\") pod \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " Oct 30 03:12:41.185647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185626 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e6408d6-96fb-4167-afb8-8db79663cedc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6408d6-96fb-4167-afb8-8db79663cedc-plugins-config-volume\") pod \"8e6408d6-96fb-4167-afb8-8db79663cedc\" (UID: \"8e6408d6-96fb-4167-afb8-8db79663cedc\") " Oct 30 03:12:41.185806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185653 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d704ffc8-e35a-448e-85a4-f665f3d8de8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d704ffc8-e35a-448e-85a4-f665f3d8de8b-plugins-config-volume\") pod \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\" (UID: \"d704ffc8-e35a-448e-85a4-f665f3d8de8b\") " Oct 30 03:12:41.185909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185892 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55qnk\" (UniqueName: \"kubernetes.io/projected/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-kube-api-access-55qnk\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.185966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185914 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.185966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185928 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.185966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.185936 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.186306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.186282 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d704ffc8-e35a-448e-85a4-f665f3d8de8b-plugins-config-volume" pod "d704ffc8-e35a-448e-85a4-f665f3d8de8b" (UID: "d704ffc8-e35a-448e-85a4-f665f3d8de8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:41.187298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.187258 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6408d6-96fb-4167-afb8-8db79663cedc-plugins-config-volume" pod "8e6408d6-96fb-4167-afb8-8db79663cedc" (UID: "8e6408d6-96fb-4167-afb8-8db79663cedc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:41.189081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.189051 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6408d6-96fb-4167-afb8-8db79663cedc-kube-api-access-jskpk" pod "8e6408d6-96fb-4167-afb8-8db79663cedc" (UID: "8e6408d6-96fb-4167-afb8-8db79663cedc"). InnerVolumeSpecName "kube-api-access-jskpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:41.189642 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.189617 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d704ffc8-e35a-448e-85a4-f665f3d8de8b-kube-api-access-kjj29" pod "d704ffc8-e35a-448e-85a4-f665f3d8de8b" (UID: "d704ffc8-e35a-448e-85a4-f665f3d8de8b"). InnerVolumeSpecName "kube-api-access-kjj29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:41.197853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.197827 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:41.287655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.287559 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6408d6-96fb-4167-afb8-8db79663cedc-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.287655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.287596 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d704ffc8-e35a-448e-85a4-f665f3d8de8b-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.287655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.287613 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jskpk\" (UniqueName: \"kubernetes.io/projected/8e6408d6-96fb-4167-afb8-8db79663cedc-kube-api-access-jskpk\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.287655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.287629 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjj29\" (UniqueName: \"kubernetes.io/projected/d704ffc8-e35a-448e-85a4-f665f3d8de8b-kube-api-access-kjj29\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:41.310891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.310808 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:41.426712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.426679 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:41.511654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.511617 4442 generic.go:356] "Generic (PLEG): container finished" podID="0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" containerID="574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23" exitCode=0 Oct 30 03:12:41.511869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.511670 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" Oct 30 03:12:41.511869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.511708 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" event={"ID":"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e","Type":"ContainerDied","Data":"574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23"} Oct 30 03:12:41.511869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.511788 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh" event={"ID":"0e36caf4-e5a8-438d-a3cd-77439dd8fd6e","Type":"ContainerDied","Data":"1d81d340c60093992e67e9ff112f5730804fcb21351a0a3873aec24096d38316"} Oct 30 03:12:41.511869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.511822 4442 scope.go:122] "RemoveContainer" containerID="574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23" Oct 30 03:12:41.513227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.513205 4442 generic.go:356] "Generic (PLEG): container finished" podID="8e6408d6-96fb-4167-afb8-8db79663cedc" containerID="71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75" exitCode=0 Oct 30 03:12:41.513365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.513262 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" Oct 30 03:12:41.513365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.513275 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" event={"ID":"8e6408d6-96fb-4167-afb8-8db79663cedc","Type":"ContainerDied","Data":"71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75"} Oct 30 03:12:41.513365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.513319 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2" event={"ID":"8e6408d6-96fb-4167-afb8-8db79663cedc","Type":"ContainerDied","Data":"dfc03c0c47877a9742cb6e6da01c141bf58816e7c87f12a8c6cfb1c440981880"} Oct 30 03:12:41.514593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.514569 4442 generic.go:356] "Generic (PLEG): container finished" podID="d704ffc8-e35a-448e-85a4-f665f3d8de8b" containerID="fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202" exitCode=0 Oct 30 03:12:41.514704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.514610 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" Oct 30 03:12:41.514704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.514650 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" event={"ID":"d704ffc8-e35a-448e-85a4-f665f3d8de8b","Type":"ContainerDied","Data":"fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202"} Oct 30 03:12:41.514704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.514694 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj" event={"ID":"d704ffc8-e35a-448e-85a4-f665f3d8de8b","Type":"ContainerDied","Data":"ea10dabd5d960477ea82625d04ba1d328abc0f3052f309d2bc76ba0c91fced85"} Oct 30 03:12:41.522928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.522913 4442 scope.go:122] "RemoveContainer" containerID="574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23" Oct 30 03:12:41.523200 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:41.523182 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23\": container with ID starting with 574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23 not found: ID does not exist" containerID="574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23" Oct 30 03:12:41.523280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.523208 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23"} err="failed to get container status \"574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23\": rpc error: code = NotFound desc = could not find container \"574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23\": container with ID starting with 574f326086d9fb605f38d4802a80e8a0b4d6bc498d47aefb17cb132152538d23 not found: ID does not exist" Oct 30 03:12:41.523280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.523226 4442 scope.go:122] "RemoveContainer" containerID="71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75" Oct 30 03:12:41.528034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.528010 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:41.532171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.532152 4442 scope.go:122] "RemoveContainer" containerID="71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75" Oct 30 03:12:41.532451 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:41.532429 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75\": container with ID starting with 71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75 not found: ID does not exist" containerID="71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75" Oct 30 03:12:41.532513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.532459 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75"} err="failed to get container status \"71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75\": rpc error: code = NotFound desc = could not find container \"71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75\": container with ID starting with 71f04b2d049f5f506cbc9847006609f29a1d54c46a9cc3f37e154e51d9b9fa75 not found: ID does not exist" Oct 30 03:12:41.532513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.532474 4442 scope.go:122] "RemoveContainer" containerID="fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202" Oct 30 03:12:41.557792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.553266 4442 scope.go:122] "RemoveContainer" containerID="fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202" Oct 30 03:12:41.557792 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:41.553702 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202\": container with ID starting with fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202 not found: ID does not exist" containerID="fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202" Oct 30 03:12:41.557792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.553738 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202"} err="failed to get container status \"fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202\": rpc error: code = NotFound desc = could not find container \"fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202\": container with ID starting with fc82e38179f355921384dc373eb838a58db6f93a000f1f4c44e244686d02c202 not found: ID does not exist" Oct 30 03:12:41.589349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.589263 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:12:41.610221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.610186 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj"] Oct 30 03:12:41.617646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.617615 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a788fe69-7dc774fc78-sj6zj"] Oct 30 03:12:41.641619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.641557 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" podUID="46f7dad2-4fb4-46f1-8fd5-03ce90420da9" containerName="payload-pre-processing" containerID="cri-o://01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a" gracePeriod=25 Oct 30 03:12:41.650181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.650161 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2"] Oct 30 03:12:41.651588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.651568 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:41.653851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.653834 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a788fe69-79c4985579-qbqv2"] Oct 30 03:12:41.687402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.687377 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh"] Oct 30 03:12:41.691875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.691853 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-acace923-5f65dcb696-f45dh"] Oct 30 03:12:41.894545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.894524 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:41.994078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.994050 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-plugins-config-volume\") pod \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " Oct 30 03:12:41.994224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.994101 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-kube-api-access-2qzz9\" (UniqueName: \"kubernetes.io/projected/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-kube-api-access-2qzz9\") pod \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\" (UID: \"46f7dad2-4fb4-46f1-8fd5-03ce90420da9\") " Oct 30 03:12:41.994427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.994404 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-plugins-config-volume" pod "46f7dad2-4fb4-46f1-8fd5-03ce90420da9" (UID: "46f7dad2-4fb4-46f1-8fd5-03ce90420da9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:41.996260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:41.996232 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-kube-api-access-2qzz9" pod "46f7dad2-4fb4-46f1-8fd5-03ce90420da9" (UID: "46f7dad2-4fb4-46f1-8fd5-03ce90420da9"). InnerVolumeSpecName "kube-api-access-2qzz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:42.043132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.043104 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:42.095453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.095426 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:42.095453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.095448 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qzz9\" (UniqueName: \"kubernetes.io/projected/46f7dad2-4fb4-46f1-8fd5-03ce90420da9-kube-api-access-2qzz9\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:42.106023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.105995 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" path="/var/lib/kubelet/pods/0e36caf4-e5a8-438d-a3cd-77439dd8fd6e/volumes" Oct 30 03:12:42.106394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.106380 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6408d6-96fb-4167-afb8-8db79663cedc" path="/var/lib/kubelet/pods/8e6408d6-96fb-4167-afb8-8db79663cedc/volumes" Oct 30 03:12:42.106728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.106716 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f94e947-d209-4059-8b0b-4dba3771cf65" path="/var/lib/kubelet/pods/8f94e947-d209-4059-8b0b-4dba3771cf65/volumes" Oct 30 03:12:42.107444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.107429 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab6add4-3b45-48a3-b8fa-3367d1879f1d" path="/var/lib/kubelet/pods/bab6add4-3b45-48a3-b8fa-3367d1879f1d/volumes" Oct 30 03:12:42.107807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.107794 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d704ffc8-e35a-448e-85a4-f665f3d8de8b" path="/var/lib/kubelet/pods/d704ffc8-e35a-448e-85a4-f665f3d8de8b/volumes" Oct 30 03:12:42.108191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.108177 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8793b3a-498e-4ff6-b81f-ce3134ce32fd" path="/var/lib/kubelet/pods/d8793b3a-498e-4ff6-b81f-ce3134ce32fd/volumes" Oct 30 03:12:42.197129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.195715 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4"] Oct 30 03:12:42.201640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.201583 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" containerID="cri-o://bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1" gracePeriod=30 Oct 30 03:12:42.212552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.212528 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:42.484407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484326 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm"] Oct 30 03:12:42.484728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484713 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d704ffc8-e35a-448e-85a4-f665f3d8de8b" containerName="payload-processing" Oct 30 03:12:42.484826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484730 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d704ffc8-e35a-448e-85a4-f665f3d8de8b" containerName="payload-processing" Oct 30 03:12:42.484826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484746 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" containerName="maas-api" Oct 30 03:12:42.484826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484770 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" containerName="maas-api" Oct 30 03:12:42.484826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484783 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c1d28d52-d3be-4499-81c7-91d885472631" containerName="payload-pre-processing" Oct 30 03:12:42.484826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484790 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d28d52-d3be-4499-81c7-91d885472631" containerName="payload-pre-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484797 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bab6add4-3b45-48a3-b8fa-3367d1879f1d" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484846 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab6add4-3b45-48a3-b8fa-3367d1879f1d" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484854 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d8793b3a-498e-4ff6-b81f-ce3134ce32fd" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484858 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8793b3a-498e-4ff6-b81f-ce3134ce32fd" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484867 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484872 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484885 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8f94e947-d209-4059-8b0b-4dba3771cf65" containerName="payload-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484891 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f94e947-d209-4059-8b0b-4dba3771cf65" containerName="payload-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484896 4442 cpu_manager.go:414] "containerMap: removing container" podUID="46f7dad2-4fb4-46f1-8fd5-03ce90420da9" containerName="payload-pre-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484901 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46f7dad2-4fb4-46f1-8fd5-03ce90420da9" containerName="payload-pre-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484909 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8e6408d6-96fb-4167-afb8-8db79663cedc" containerName="payload-pre-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484913 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6408d6-96fb-4167-afb8-8db79663cedc" containerName="payload-pre-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484970 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d704ffc8-e35a-448e-85a4-f665f3d8de8b" containerName="payload-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484982 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e36caf4-e5a8-438d-a3cd-77439dd8fd6e" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484989 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d28d52-d3be-4499-81c7-91d885472631" containerName="payload-pre-processing" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484994 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab6add4-3b45-48a3-b8fa-3367d1879f1d" containerName="maas-api" Oct 30 03:12:42.484993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.484999 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8793b3a-498e-4ff6-b81f-ce3134ce32fd" containerName="maas-api" Oct 30 03:12:42.485432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.485004 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c38b5a6-41c2-4d0f-955a-2c4a4b80e735" containerName="maas-api" Oct 30 03:12:42.485432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.485010 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f94e947-d209-4059-8b0b-4dba3771cf65" containerName="payload-processing" Oct 30 03:12:42.485432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.485015 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="46f7dad2-4fb4-46f1-8fd5-03ce90420da9" containerName="payload-pre-processing" Oct 30 03:12:42.485432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.485020 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6408d6-96fb-4167-afb8-8db79663cedc" containerName="payload-pre-processing" Oct 30 03:12:42.489475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.489455 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.493654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.493627 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:42.509693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.509663 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm"] Oct 30 03:12:42.515243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.515222 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2d5b09-openshift-default-dockercfg-hpf84\"" Oct 30 03:12:42.524736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.524710 4442 generic.go:356] "Generic (PLEG): container finished" podID="46f7dad2-4fb4-46f1-8fd5-03ce90420da9" containerID="01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a" exitCode=0 Oct 30 03:12:42.524880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.524786 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" Oct 30 03:12:42.524880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.524785 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" event={"ID":"46f7dad2-4fb4-46f1-8fd5-03ce90420da9","Type":"ContainerDied","Data":"01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a"} Oct 30 03:12:42.524992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.524910 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk" event={"ID":"46f7dad2-4fb4-46f1-8fd5-03ce90420da9","Type":"ContainerDied","Data":"0fdbc8d3e68eaa98ccdf55e3e5c7d61a09f5e9d05c5e9b59188ae6b54edd1d13"} Oct 30 03:12:42.524992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.524936 4442 scope.go:122] "RemoveContainer" containerID="01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a" Oct 30 03:12:42.537466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.537433 4442 scope.go:122] "RemoveContainer" containerID="01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a" Oct 30 03:12:42.537719 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:42.537695 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a\": container with ID starting with 01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a not found: ID does not exist" containerID="01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a" Oct 30 03:12:42.537813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.537726 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a"} err="failed to get container status \"01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a\": rpc error: code = NotFound desc = could not find container \"01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a\": container with ID starting with 01fb5da8f2746f48606565d440fad82c23df99e4707875645b6a83400fcabc1a not found: ID does not exist" Oct 30 03:12:42.570550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.570526 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk"] Oct 30 03:12:42.577845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.577824 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c7c56c3d-6b99fb575-96bgk"] Oct 30 03:12:42.600428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600386 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.600649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600439 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.600649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600486 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.600649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600515 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.600649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600553 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-694d7\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-kube-api-access-694d7\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.600649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600583 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.600649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600645 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.601014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600674 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.601014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.600739 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f428cce2-c5cc-4903-bbeb-898b1a84d249-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.652492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.652451 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:42.701951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.701918 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.701951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.701949 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.701977 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702005 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702034 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-694d7\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-kube-api-access-694d7\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702064 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702119 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702288 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702356 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f428cce2-c5cc-4903-bbeb-898b1a84d249-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702424 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-certs\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702539 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-data\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702567 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-credential-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.702884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702704 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-socket\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.703011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.702994 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f428cce2-c5cc-4903-bbeb-898b1a84d249-istiod-ca-cert\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.704540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.704520 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-envoy\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.704700 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.704670 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-podinfo\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.720870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.720846 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-694d7\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-kube-api-access-694d7\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.721070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.721050 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-token\") pod \"e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.803525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.803490 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:42.823879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.823845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:42.962507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.962475 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p"] Oct 30 03:12:42.966868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.966842 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:42.984283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.984240 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3c874f-openshift-default-dockercfg-zt77d\"" Oct 30 03:12:42.987354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:42.987330 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p"] Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009256 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-data\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009313 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-certs\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009338 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107b2e13-5fb2-4241-8143-e75258d44261-istio-podinfo\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009364 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-istio-token\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009404 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-credential-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009443 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107b2e13-5fb2-4241-8143-e75258d44261-istiod-ca-cert\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009471 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-envoy\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009515 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.009782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.009562 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84d4\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-kube-api-access-n84d4\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.022340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.022297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:43.074867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.074774 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj"] Oct 30 03:12:43.075037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.075006 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" podUID="d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" containerName="maas-api" containerID="cri-o://51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624" gracePeriod=30 Oct 30 03:12:43.110866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.110835 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.110987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.110894 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84d4\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-kube-api-access-n84d4\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.110987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.110979 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-data\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.111112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111020 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-certs\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.111112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111049 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107b2e13-5fb2-4241-8143-e75258d44261-istio-podinfo\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.111112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111075 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-istio-token\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.111257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111121 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-credential-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.111257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111166 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107b2e13-5fb2-4241-8143-e75258d44261-istiod-ca-cert\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.111257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111198 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-envoy\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.111257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111248 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.116821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111470 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-data\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.116821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111687 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-credential-socket\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.116821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.111705 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-certs\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.116821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.112979 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107b2e13-5fb2-4241-8143-e75258d44261-istiod-ca-cert\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.116821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.114025 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-envoy\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.116821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.114355 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107b2e13-5fb2-4241-8143-e75258d44261-istio-podinfo\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.116821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.115015 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm"] Oct 30 03:12:43.130234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.130209 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84d4\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-kube-api-access-n84d4\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.130594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.130572 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-istio-token\") pod \"e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.139686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.139663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:43.146189 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:43.146166 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf428cce2_c5cc_4903_bbeb_898b1a84d249.slice/crio-6b64f6e62a7c793dd86b3094937db8d961bf4c0c3e604babfe7184886b229668 WatchSource:0}: Error finding container 6b64f6e62a7c793dd86b3094937db8d961bf4c0c3e604babfe7184886b229668: Status 404 returned error can't find the container with id 6b64f6e62a7c793dd86b3094937db8d961bf4c0c3e604babfe7184886b229668 Oct 30 03:12:43.181736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.181695 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:43.181854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.181814 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:43.181925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.181907 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:43.257671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.250393 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:43.280928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.280891 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:43.316042 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.316018 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:43.413570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.413534 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-maas-api-tls\") pod \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " Oct 30 03:12:43.413736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.413647 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-openshift-service-ca\") pod \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " Oct 30 03:12:43.413815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.413793 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-metrics-certs\") pod \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " Oct 30 03:12:43.413860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.413830 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-kube-api-access-tfc8z\" (UniqueName: \"kubernetes.io/projected/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-kube-api-access-tfc8z\") pod \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\" (UID: \"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd\") " Oct 30 03:12:43.414072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.414042 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-openshift-service-ca" pod "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" (UID: "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:43.414175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.414156 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:43.416061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.416021 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-maas-api-tls" pod "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" (UID: "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:43.416276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.416254 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-kube-api-access-tfc8z" pod "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" (UID: "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd"). InnerVolumeSpecName "kube-api-access-tfc8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:43.416469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.416444 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-metrics-certs" pod "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" (UID: "d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:43.467028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.466994 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp"] Oct 30 03:12:43.467589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.467565 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" containerName="maas-api" Oct 30 03:12:43.467703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.467591 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" containerName="maas-api" Oct 30 03:12:43.467793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.467719 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" containerName="maas-api" Oct 30 03:12:43.471412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.471392 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.489820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.489792 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8d80b1-openshift-default-dockercfg-22948\"" Oct 30 03:12:43.492413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.492378 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp"] Oct 30 03:12:43.515599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.515574 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:43.515599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.515599 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfc8z\" (UniqueName: \"kubernetes.io/projected/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-kube-api-access-tfc8z\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:43.515772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.515611 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:43.520746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.520717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:43.534413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.534380 4442 generic.go:356] "Generic (PLEG): container finished" podID="d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" containerID="51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624" exitCode=0 Oct 30 03:12:43.534542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.534433 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" Oct 30 03:12:43.534542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.534464 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" event={"ID":"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd","Type":"ContainerDied","Data":"51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624"} Oct 30 03:12:43.534542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.534520 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj" event={"ID":"d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd","Type":"ContainerDied","Data":"6489e5ac1521b324529150855bdad21a498d6efbfd85ae32cf02d747a0767526"} Oct 30 03:12:43.534542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.534543 4442 scope.go:122] "RemoveContainer" containerID="51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624" Oct 30 03:12:43.536541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.536514 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" event={"ID":"f428cce2-c5cc-4903-bbeb-898b1a84d249","Type":"ContainerStarted","Data":"e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb"} Oct 30 03:12:43.536686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.536572 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" event={"ID":"f428cce2-c5cc-4903-bbeb-898b1a84d249","Type":"ContainerStarted","Data":"6b64f6e62a7c793dd86b3094937db8d961bf4c0c3e604babfe7184886b229668"} Oct 30 03:12:43.546773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.546741 4442 scope.go:122] "RemoveContainer" containerID="51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624" Oct 30 03:12:43.547118 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:43.547098 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624\": container with ID starting with 51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624 not found: ID does not exist" containerID="51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624" Oct 30 03:12:43.547211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.547126 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624"} err="failed to get container status \"51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624\": rpc error: code = NotFound desc = could not find container \"51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624\": container with ID starting with 51d821dfc78e7907cfeb85d28e68bb62614c37d664428e3433ee38b15278e624 not found: ID does not exist" Oct 30 03:12:43.590375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.586840 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj"] Oct 30 03:12:43.590375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.589823 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3f9206f0-868544db57-xz9tj"] Oct 30 03:12:43.597652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.597580 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p"] Oct 30 03:12:43.613020 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:43.612985 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107b2e13_5fb2_4241_8143_e75258d44261.slice/crio-e0c4dc6bcc1e31411070ded8b46166b3aa3a9968eae4d340d24d26f31e00fb10 WatchSource:0}: Error finding container e0c4dc6bcc1e31411070ded8b46166b3aa3a9968eae4d340d24d26f31e00fb10: Status 404 returned error can't find the container with id e0c4dc6bcc1e31411070ded8b46166b3aa3a9968eae4d340d24d26f31e00fb10 Oct 30 03:12:43.617504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.617465 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:43.617629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.617537 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:43.617629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.617566 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:43.618557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618534 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618601 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618639 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618660 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f81e79-58a2-40e4-aa2a-464b18728733-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618682 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618704 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvc5g\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-kube-api-access-vvc5g\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618721 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618742 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f81e79-58a2-40e4-aa2a-464b18728733-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.618889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.618855 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.636076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.636009 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podStartSLOduration=1.6359899279999999 podStartE2EDuration="1.635989928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:43.634910887 +0000 UTC m=+587.637264087" watchObservedRunningTime="2026-10-30 03:12:43.635989928 +0000 UTC m=+587.638343121" Oct 30 03:12:43.719876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.719845 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.719894 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.719936 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.719983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720017 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f81e79-58a2-40e4-aa2a-464b18728733-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720050 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720082 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvc5g\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-kube-api-access-vvc5g\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720108 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720144 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f81e79-58a2-40e4-aa2a-464b18728733-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720294 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720363 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720868 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720883 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f81e79-58a2-40e4-aa2a-464b18728733-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.720919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.720938 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.722454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.722436 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.722679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.722663 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f81e79-58a2-40e4-aa2a-464b18728733-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.767702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.763271 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvc5g\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-kube-api-access-vvc5g\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.769300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.769242 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.779994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.779965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:43.787031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.787000 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:43.804217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.804197 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:43.918247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:43.918211 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:44.050168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.050134 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:44.106277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.106209 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46f7dad2-4fb4-46f1-8fd5-03ce90420da9" path="/var/lib/kubelet/pods/46f7dad2-4fb4-46f1-8fd5-03ce90420da9/volumes" Oct 30 03:12:44.106657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.106643 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd" path="/var/lib/kubelet/pods/d1e36b67-caac-4cd1-8a80-9a2c03a5e8dd/volumes" Oct 30 03:12:44.139399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.139374 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp"] Oct 30 03:12:44.141404 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:44.141373 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f81e79_58a2_40e4_aa2a_464b18728733.slice/crio-ebb6d7d68dee06d6b325d46eb70e9d785c930baf986903ee441fcffa40f12c17 WatchSource:0}: Error finding container ebb6d7d68dee06d6b325d46eb70e9d785c930baf986903ee441fcffa40f12c17: Status 404 returned error can't find the container with id ebb6d7d68dee06d6b325d46eb70e9d785c930baf986903ee441fcffa40f12c17 Oct 30 03:12:44.144182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.144145 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:44.144264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.144233 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:44.144300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.144276 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:12:44.544281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.544249 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" event={"ID":"70f81e79-58a2-40e4-aa2a-464b18728733","Type":"ContainerStarted","Data":"84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9"} Oct 30 03:12:44.544474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.544305 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" event={"ID":"70f81e79-58a2-40e4-aa2a-464b18728733","Type":"ContainerStarted","Data":"ebb6d7d68dee06d6b325d46eb70e9d785c930baf986903ee441fcffa40f12c17"} Oct 30 03:12:44.545804 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.545776 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" event={"ID":"107b2e13-5fb2-4241-8143-e75258d44261","Type":"ContainerStarted","Data":"d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16"} Oct 30 03:12:44.545920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.545834 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" event={"ID":"107b2e13-5fb2-4241-8143-e75258d44261","Type":"ContainerStarted","Data":"e0c4dc6bcc1e31411070ded8b46166b3aa3a9968eae4d340d24d26f31e00fb10"} Oct 30 03:12:44.606521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.606442 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podStartSLOduration=1.6064283609999999 podStartE2EDuration="1.606428361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:44.604401825 +0000 UTC m=+588.606755024" watchObservedRunningTime="2026-10-30 03:12:44.606428361 +0000 UTC m=+588.608781552" Oct 30 03:12:44.645961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.645892 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:44.665253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.665208 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" podStartSLOduration=2.665196991 podStartE2EDuration="2.665196991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:44.664463613 +0000 UTC m=+588.666816821" watchObservedRunningTime="2026-10-30 03:12:44.665196991 +0000 UTC m=+588.667550189" Oct 30 03:12:44.746085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.746055 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:44.788172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.788149 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:44.819506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.819412 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:44.952018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:44.951982 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:45.281554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.281522 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:45.286684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.286661 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:45.551175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.550797 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:45.551707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.551678 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:45.552025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.552004 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:12:45.552194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.552182 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:12:45.629177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.629147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:45.742542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.742511 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:45.748512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.748482 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d"] Oct 30 03:12:45.751916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.751901 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.771519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.771494 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a9c9b6-serving-cert\"" Oct 30 03:12:45.779097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.779065 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d"] Oct 30 03:12:45.793087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.793059 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:45.842559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.842474 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v986p\" (UniqueName: \"kubernetes.io/projected/4ef1519e-576f-41a0-9a11-9596736afc1b-kube-api-access-v986p\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.842706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.842601 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef1519e-576f-41a0-9a11-9596736afc1b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.842706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.842669 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.842863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.842822 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-metrics-certs\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.845156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.845110 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:45.888888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.888862 4442 patch_prober.go:29] interesting pod/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:12:45.889030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.888920 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:12:45.950780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.949331 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.950780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.949417 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-metrics-certs\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.950780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.949512 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v986p\" (UniqueName: \"kubernetes.io/projected/4ef1519e-576f-41a0-9a11-9596736afc1b-kube-api-access-v986p\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.950780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.949612 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef1519e-576f-41a0-9a11-9596736afc1b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.950780 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:45.950145 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-serving-cert: secret "maas-api-e2e-worker-w0-a9c9b6-serving-cert" not found Oct 30 03:12:45.950780 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:45.950226 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls podName:4ef1519e-576f-41a0-9a11-9596736afc1b nodeName:}" failed. No retries permitted until 2026-10-30 03:12:46.4502062 +0000 UTC m=+590.452559402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls") pod "maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" (UID: "4ef1519e-576f-41a0-9a11-9596736afc1b") : secret "maas-api-e2e-worker-w0-a9c9b6-serving-cert" not found Oct 30 03:12:45.950780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.950452 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:45.968775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.958612 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef1519e-576f-41a0-9a11-9596736afc1b-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.981679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.980287 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-metrics-certs\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:45.993393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:45.991411 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v986p\" (UniqueName: \"kubernetes.io/projected/4ef1519e-576f-41a0-9a11-9596736afc1b-kube-api-access-v986p\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:46.140783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.140652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:46.246020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.245986 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:46.347114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.347077 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:46.449540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.449369 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:46.457298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.457249 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:46.472411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.472377 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:46.557951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.557920 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:46.562397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.562375 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:12:46.664905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.664878 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:46.965738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:46.965689 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d"] Oct 30 03:12:46.967107 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:46.967075 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef1519e_576f_41a0_9a11_9596736afc1b.slice/crio-7730fc5cebbf34405dc72e4eca7a53bb04a71b1f875b57968e4efe958d29d852 WatchSource:0}: Error finding container 7730fc5cebbf34405dc72e4eca7a53bb04a71b1f875b57968e4efe958d29d852: Status 404 returned error can't find the container with id 7730fc5cebbf34405dc72e4eca7a53bb04a71b1f875b57968e4efe958d29d852 Oct 30 03:12:47.214891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.214862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:47.342409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.342379 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:47.393014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.392977 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:12:47.461065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.461032 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:12:47.562051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.562023 4442 generic.go:356] "Generic (PLEG): container finished" podID="efabf41c-6966-428b-a148-4841fecfbfa3" containerID="bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1" exitCode=0 Oct 30 03:12:47.562212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.562076 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" Oct 30 03:12:47.562212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.562092 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" event={"ID":"efabf41c-6966-428b-a148-4841fecfbfa3","Type":"ContainerDied","Data":"bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1"} Oct 30 03:12:47.562212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.562142 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4" event={"ID":"efabf41c-6966-428b-a148-4841fecfbfa3","Type":"ContainerDied","Data":"e1d7f6591b279aad54a150f75edaf5987b34468c7a0fad26f3b3a2666a1cce8c"} Oct 30 03:12:47.562212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.562173 4442 scope.go:122] "RemoveContainer" containerID="bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1" Oct 30 03:12:47.563734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.563705 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" event={"ID":"4ef1519e-576f-41a0-9a11-9596736afc1b","Type":"ContainerStarted","Data":"27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b"} Oct 30 03:12:47.563879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.563773 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" event={"ID":"4ef1519e-576f-41a0-9a11-9596736afc1b","Type":"ContainerStarted","Data":"7730fc5cebbf34405dc72e4eca7a53bb04a71b1f875b57968e4efe958d29d852"} Oct 30 03:12:47.563959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.563922 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:47.571217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.571190 4442 scope.go:122] "RemoveContainer" containerID="db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859" Oct 30 03:12:47.574306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574286 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-credential-socket\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574329 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efabf41c-6966-428b-a148-4841fecfbfa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efabf41c-6966-428b-a148-4841fecfbfa3-istiod-ca-cert\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574365 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-data\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574413 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-kube-api-access-m6mwd\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-kube-api-access-m6mwd\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574451 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-istio-token\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-istio-token\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574507 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-certs\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574530 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-credential-socket" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:12:47.574664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574544 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efabf41c-6966-428b-a148-4841fecfbfa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efabf41c-6966-428b-a148-4841fecfbfa3-istio-podinfo\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574606 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-data" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:12:47.574664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574620 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-envoy\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574650 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-socket\") pod \"efabf41c-6966-428b-a148-4841fecfbfa3\" (UID: \"efabf41c-6966-428b-a148-4841fecfbfa3\") " Oct 30 03:12:47.574901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574716 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efabf41c-6966-428b-a148-4841fecfbfa3-istiod-ca-cert" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:47.575013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574962 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.575013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574986 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efabf41c-6966-428b-a148-4841fecfbfa3-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.575013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.575000 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.575013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.574981 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-certs" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:12:47.575456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.575131 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-socket" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:12:47.576896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.576871 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efabf41c-6966-428b-a148-4841fecfbfa3-istio-podinfo" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:12:47.576974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.576950 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-kube-api-access-m6mwd" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "kube-api-access-m6mwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:47.577312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.577291 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-envoy" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:12:47.577394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.577371 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-istio-token" pod "efabf41c-6966-428b-a148-4841fecfbfa3" (UID: "efabf41c-6966-428b-a148-4841fecfbfa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:47.580285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.580263 4442 scope.go:122] "RemoveContainer" containerID="bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1" Oct 30 03:12:47.580535 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:47.580514 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1\": container with ID starting with bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1 not found: ID does not exist" containerID="bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1" Oct 30 03:12:47.580587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.580542 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1"} err="failed to get container status \"bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1\": rpc error: code = NotFound desc = could not find container \"bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1\": container with ID starting with bb7005320669778b7883080a8f20068a569e5ac80e8e52bd0004d7819dba3ab1 not found: ID does not exist" Oct 30 03:12:47.580587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.580558 4442 scope.go:122] "RemoveContainer" containerID="db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859" Oct 30 03:12:47.580833 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:47.580815 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859\": container with ID starting with db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859 not found: ID does not exist" containerID="db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859" Oct 30 03:12:47.580879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.580840 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859"} err="failed to get container status \"db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859\": rpc error: code = NotFound desc = could not find container \"db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859\": container with ID starting with db390f9e837f1dbfd4e31a0169c6e63b1db39ca672f65c7502adb59735264859 not found: ID does not exist" Oct 30 03:12:47.620977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.620794 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" podStartSLOduration=2.298381494 podStartE2EDuration="2.620779694s" totalImagesPullingTime="322.3982ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:47.620001048 +0000 UTC m=+591.622354273" watchObservedRunningTime="2026-10-30 03:12:47.620779694 +0000 UTC m=+591.623132888" Oct 30 03:12:47.676590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.676557 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6mwd\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-kube-api-access-m6mwd\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.676590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.676590 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efabf41c-6966-428b-a148-4841fecfbfa3-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.676590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.676619 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.676880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.676632 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efabf41c-6966-428b-a148-4841fecfbfa3-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.676880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.676644 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.676880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.676658 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efabf41c-6966-428b-a148-4841fecfbfa3-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:12:47.904794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.904739 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4"] Oct 30 03:12:47.915374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:47.915346 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c7c56c3d-openshift-default-7b54b69b4f-s5lr4"] Oct 30 03:12:48.012592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.012561 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:48.085326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.085294 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh"] Oct 30 03:12:48.085722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.085708 4442 cpu_manager.go:414] "containerMap: removing container" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" Oct 30 03:12:48.085815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.085726 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" Oct 30 03:12:48.085815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.085745 4442 cpu_manager.go:414] "containerMap: removing container" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" Oct 30 03:12:48.085815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.085767 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" Oct 30 03:12:48.085934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.085836 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" Oct 30 03:12:48.085934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.085852 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" containerName="istio-proxy" Oct 30 03:12:48.090272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.090255 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.105817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.105783 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efabf41c-6966-428b-a148-4841fecfbfa3" path="/var/lib/kubelet/pods/efabf41c-6966-428b-a148-4841fecfbfa3/volumes" Oct 30 03:12:48.124608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.124533 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-2d5b09-serving-cert\"" Oct 30 03:12:48.127048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.127023 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh"] Oct 30 03:12:48.139381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.139353 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:48.182143 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.182117 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zvx\" (UniqueName: \"kubernetes.io/projected/7f2295a7-6365-4457-8278-7bd0c1f3511c-kube-api-access-p9zvx\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.182291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.182177 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.182440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.182421 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-metrics-certs\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.182521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.182506 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2295a7-6365-4457-8278-7bd0c1f3511c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.283406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.283370 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-metrics-certs\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.283707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.283681 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2295a7-6365-4457-8278-7bd0c1f3511c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.283829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.283806 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zvx\" (UniqueName: \"kubernetes.io/projected/7f2295a7-6365-4457-8278-7bd0c1f3511c-kube-api-access-p9zvx\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.283907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.283850 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.284663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.284640 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2295a7-6365-4457-8278-7bd0c1f3511c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.286213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.286190 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-metrics-certs\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.286796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.286746 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.287655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.287612 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:48.307262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.307238 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zvx\" (UniqueName: \"kubernetes.io/projected/7f2295a7-6365-4457-8278-7bd0c1f3511c-kube-api-access-p9zvx\") pod \"maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.398406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.398323 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:48.403332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.403306 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:48.504704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.504388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:48.713011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:48.712983 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh"] Oct 30 03:12:48.715558 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:48.715535 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2295a7_6365_4457_8278_7bd0c1f3511c.slice/crio-ab5a9f4b7db990f48792478eb10c2ddb1f7b15665f2942b0a51298ec7a1d46cf WatchSource:0}: Error finding container ab5a9f4b7db990f48792478eb10c2ddb1f7b15665f2942b0a51298ec7a1d46cf: Status 404 returned error can't find the container with id ab5a9f4b7db990f48792478eb10c2ddb1f7b15665f2942b0a51298ec7a1d46cf Oct 30 03:12:49.575290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:49.575250 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" event={"ID":"7f2295a7-6365-4457-8278-7bd0c1f3511c","Type":"ContainerStarted","Data":"a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543"} Oct 30 03:12:49.575695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:49.575325 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" event={"ID":"7f2295a7-6365-4457-8278-7bd0c1f3511c","Type":"ContainerStarted","Data":"ab5a9f4b7db990f48792478eb10c2ddb1f7b15665f2942b0a51298ec7a1d46cf"} Oct 30 03:12:49.575695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:49.575347 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:49.625950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:49.625890 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" podStartSLOduration=1.188840729 podStartE2EDuration="1.625874133s" totalImagesPullingTime="437.033404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:49.625861186 +0000 UTC m=+593.628214402" watchObservedRunningTime="2026-10-30 03:12:49.625874133 +0000 UTC m=+593.628227326" Oct 30 03:12:50.222582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:50.222548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:50.525746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:50.525716 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:50.634184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:50.634154 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:50.736819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:50.736788 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:51.506188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:51.506161 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:51.609311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:51.609283 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:52.112480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.112451 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4"] Oct 30 03:12:52.116473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.116452 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.130535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.130516 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-3c874f-serving-cert\"" Oct 30 03:12:52.144824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.144784 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4"] Oct 30 03:12:52.193827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.193781 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:52.221648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.221621 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57bjb\" (UniqueName: \"kubernetes.io/projected/5c36d571-4774-49b7-8e1e-25821eb776d5-kube-api-access-57bjb\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.221817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.221663 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c36d571-4774-49b7-8e1e-25821eb776d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.221817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.221684 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.221933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.221853 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-metrics-certs\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.289266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.289244 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:52.323263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.323106 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57bjb\" (UniqueName: \"kubernetes.io/projected/5c36d571-4774-49b7-8e1e-25821eb776d5-kube-api-access-57bjb\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.323263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.323182 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c36d571-4774-49b7-8e1e-25821eb776d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.323263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.323216 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.323588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.323281 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-metrics-certs\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.324265 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:52.324235 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-serving-cert: secret "maas-api-e2e-worker-w3-3c874f-serving-cert" not found Oct 30 03:12:52.324467 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:52.324307 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls podName:5c36d571-4774-49b7-8e1e-25821eb776d5 nodeName:}" failed. No retries permitted until 2026-10-30 03:12:52.824288566 +0000 UTC m=+596.826641754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls") pod "maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" (UID: "5c36d571-4774-49b7-8e1e-25821eb776d5") : secret "maas-api-e2e-worker-w3-3c874f-serving-cert" not found Oct 30 03:12:52.324552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.324499 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c36d571-4774-49b7-8e1e-25821eb776d5-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.328632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.328608 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-metrics-certs\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.344260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.344238 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57bjb\" (UniqueName: \"kubernetes.io/projected/5c36d571-4774-49b7-8e1e-25821eb776d5-kube-api-access-57bjb\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.468997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.468913 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:52.827989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.827955 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:52.830380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:52.830352 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:53.029659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:53.029619 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:53.328037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:53.328013 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4"] Oct 30 03:12:53.329992 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:53.329931 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c36d571_4774_49b7_8e1e_25821eb776d5.slice/crio-f8422dc16d98798052cab53d4b5fb808259f73678e5c60dc6dab45433fc8a536 WatchSource:0}: Error finding container f8422dc16d98798052cab53d4b5fb808259f73678e5c60dc6dab45433fc8a536: Status 404 returned error can't find the container with id f8422dc16d98798052cab53d4b5fb808259f73678e5c60dc6dab45433fc8a536 Oct 30 03:12:53.574690 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:53.574658 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 03:12:53.596872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:53.596793 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" event={"ID":"5c36d571-4774-49b7-8e1e-25821eb776d5","Type":"ContainerStarted","Data":"f8422dc16d98798052cab53d4b5fb808259f73678e5c60dc6dab45433fc8a536"} Oct 30 03:12:54.041839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.041806 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:54.117176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.117145 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9"] Oct 30 03:12:54.121317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.121296 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.142480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.142452 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9"] Oct 30 03:12:54.143022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.142999 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8d80b1-serving-cert\"" Oct 30 03:12:54.147558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.147530 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:54.241940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.241912 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.242118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.241950 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b80988-fbb8-4d18-910b-190fc3c6bcdd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.242178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.242147 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-metrics-certs\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.242216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.242178 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68dwv\" (UniqueName: \"kubernetes.io/projected/14b80988-fbb8-4d18-910b-190fc3c6bcdd-kube-api-access-68dwv\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.249579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.249554 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:54.343500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.343400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-metrics-certs\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.343500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.343475 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68dwv\" (UniqueName: \"kubernetes.io/projected/14b80988-fbb8-4d18-910b-190fc3c6bcdd-kube-api-access-68dwv\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.344057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.343575 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.344057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.343600 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b80988-fbb8-4d18-910b-190fc3c6bcdd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.345336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.345316 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b80988-fbb8-4d18-910b-190fc3c6bcdd-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.350041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.350010 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:54.350231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.350088 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-metrics-certs\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.351472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.351445 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.376428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.368484 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68dwv\" (UniqueName: \"kubernetes.io/projected/14b80988-fbb8-4d18-910b-190fc3c6bcdd-kube-api-access-68dwv\") pod \"maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.434179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.434149 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:54.447194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.447149 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:54.612875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.612776 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" event={"ID":"5c36d571-4774-49b7-8e1e-25821eb776d5","Type":"ContainerStarted","Data":"9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047"} Oct 30 03:12:54.613059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.612892 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:12:54.755805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.755737 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" podStartSLOduration=2.432547251 podStartE2EDuration="2.755722299s" totalImagesPullingTime="323.175048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:54.663174149 +0000 UTC m=+598.665527347" watchObservedRunningTime="2026-10-30 03:12:54.755722299 +0000 UTC m=+598.758075496" Oct 30 03:12:54.756394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:54.756374 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9"] Oct 30 03:12:54.757685 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:12:54.757662 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b80988_fbb8_4d18_910b_190fc3c6bcdd.slice/crio-7b561d51d9fdd96d403cff1bb589ed9de685801ed4e3d6ff560de839458e55d3 WatchSource:0}: Error finding container 7b561d51d9fdd96d403cff1bb589ed9de685801ed4e3d6ff560de839458e55d3: Status 404 returned error can't find the container with id 7b561d51d9fdd96d403cff1bb589ed9de685801ed4e3d6ff560de839458e55d3 Oct 30 03:12:54.866511 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:12:54.866425 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:12:55.607071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:55.607039 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 03:12:55.620193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:55.620159 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" event={"ID":"14b80988-fbb8-4d18-910b-190fc3c6bcdd","Type":"ContainerStarted","Data":"627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874"} Oct 30 03:12:55.620382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:55.620223 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" event={"ID":"14b80988-fbb8-4d18-910b-190fc3c6bcdd","Type":"ContainerStarted","Data":"7b561d51d9fdd96d403cff1bb589ed9de685801ed4e3d6ff560de839458e55d3"} Oct 30 03:12:55.620382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:55.620324 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:12:55.711795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:55.710863 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" podStartSLOduration=1.360847672 podStartE2EDuration="1.710840518s" totalImagesPullingTime="349.992846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:12:55.709145014 +0000 UTC m=+599.711498213" watchObservedRunningTime="2026-10-30 03:12:55.710840518 +0000 UTC m=+599.713193718" Oct 30 03:12:56.151770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:56.151727 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:12:56.158667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:56.158644 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:12:56.246658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:56.246628 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:58.101358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:58.101303 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:58.707781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:58.704248 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:12:58.720399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:58.720369 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:59.376263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:59.376232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:59.585252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:59.585216 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:12:59.704552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:12:59.704464 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:00.283415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.283005 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6"] Oct 30 03:13:00.290880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.290853 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.309128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.308418 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-e44313-serving-cert\"" Oct 30 03:13:00.336288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.336253 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6"] Oct 30 03:13:00.431777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.431256 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:00.432279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.431988 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38351d05-2730-4456-8bb6-46226286b4c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.432279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.432084 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hzs4\" (UniqueName: \"kubernetes.io/projected/38351d05-2730-4456-8bb6-46226286b4c1-kube-api-access-7hzs4\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.432279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.432121 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-metrics-certs\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.432279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.432183 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.542131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.540313 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38351d05-2730-4456-8bb6-46226286b4c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.542131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.540428 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hzs4\" (UniqueName: \"kubernetes.io/projected/38351d05-2730-4456-8bb6-46226286b4c1-kube-api-access-7hzs4\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.542131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.540460 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-metrics-certs\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.542131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.540528 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.542131 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:13:00.540704 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-serving-cert: secret "maas-api-e2e-worker-w1-e44313-serving-cert" not found Oct 30 03:13:00.542131 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:13:00.540811 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls podName:38351d05-2730-4456-8bb6-46226286b4c1 nodeName:}" failed. No retries permitted until 2026-10-30 03:13:01.040786612 +0000 UTC m=+605.043139809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls") pod "maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" (UID: "38351d05-2730-4456-8bb6-46226286b4c1") : secret "maas-api-e2e-worker-w1-e44313-serving-cert" not found Oct 30 03:13:00.542131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.541804 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38351d05-2730-4456-8bb6-46226286b4c1-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.549695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.547401 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-metrics-certs\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.559640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.559606 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:00.620800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.620470 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hzs4\" (UniqueName: \"kubernetes.io/projected/38351d05-2730-4456-8bb6-46226286b4c1-kube-api-access-7hzs4\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:00.646809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.646744 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 03:13:00.764637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.764565 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:00.964600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:00.964485 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:01.060026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.059555 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:01.062464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.062437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls\") pod \"maas-api-e2e-worker-w1-e44313-587c745445-jg7g6\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:01.258021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.257991 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:01.595308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.594869 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:13:01.595308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.594941 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:13:01.628736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.628610 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:01.638902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.638862 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6"] Oct 30 03:13:01.645207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.645184 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 03:13:01.645975 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:13:01.645939 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38351d05_2730_4456_8bb6_46226286b4c1.slice/crio-4d1ade95c9e449333acb5b16c3d447d0c2e1e3bc158b09d271aa246dc53bd2f3 WatchSource:0}: Error finding container 4d1ade95c9e449333acb5b16c3d447d0c2e1e3bc158b09d271aa246dc53bd2f3: Status 404 returned error can't find the container with id 4d1ade95c9e449333acb5b16c3d447d0c2e1e3bc158b09d271aa246dc53bd2f3 Oct 30 03:13:01.655383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.655354 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" event={"ID":"38351d05-2730-4456-8bb6-46226286b4c1","Type":"ContainerStarted","Data":"4d1ade95c9e449333acb5b16c3d447d0c2e1e3bc158b09d271aa246dc53bd2f3"} Oct 30 03:13:01.737060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.736621 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:01.840935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:01.840810 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:02.039743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:02.039531 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:02.662434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:02.662397 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" event={"ID":"38351d05-2730-4456-8bb6-46226286b4c1","Type":"ContainerStarted","Data":"7ccba4f0e541cdde905d323f2675d1d857991e2409ade61ca7c87f05844d7899"} Oct 30 03:13:02.662954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:02.662464 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:03.058068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:03.058019 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:05.131244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.131200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:05.218590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.218467 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" podStartSLOduration=4.893180688 podStartE2EDuration="5.218444908s" totalImagesPullingTime="325.26422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:02.716283175 +0000 UTC m=+606.718636386" watchObservedRunningTime="2026-10-30 03:13:05.218444908 +0000 UTC m=+609.220798107" Oct 30 03:13:05.219614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.219582 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:13:05.685640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.685587 4442 generic.go:356] "Generic (PLEG): container finished" podID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerID="e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb" exitCode=0 Oct 30 03:13:05.686081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.686057 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" event={"ID":"f428cce2-c5cc-4903-bbeb-898b1a84d249","Type":"ContainerDied","Data":"e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb"} Oct 30 03:13:05.687100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.686988 4442 scope.go:122] "RemoveContainer" containerID="e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb" Oct 30 03:13:05.690222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.690189 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:05.690536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.690514 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:05.690724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:05.690694 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:06.714444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:06.714410 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" event={"ID":"f428cce2-c5cc-4903-bbeb-898b1a84d249","Type":"ContainerStarted","Data":"f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357"} Oct 30 03:13:06.807081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:06.806944 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:13:06.807264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:06.807199 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 03:13:06.807264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:06.807242 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 03:13:07.280387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:07.280346 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:07.807779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:07.805152 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 03:13:07.807779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:07.805217 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 03:13:08.673668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:08.673641 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 03:13:08.804930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:08.804419 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 03:13:08.804930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:08.804476 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 03:13:09.279990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:09.279913 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:09.810343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:09.805153 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 03:13:09.810343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:09.805211 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 03:13:10.737575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.737545 4442 generic.go:356] "Generic (PLEG): container finished" podID="107b2e13-5fb2-4241-8143-e75258d44261" containerID="d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16" exitCode=0 Oct 30 03:13:10.738092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.737620 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" event={"ID":"107b2e13-5fb2-4241-8143-e75258d44261","Type":"ContainerDied","Data":"d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16"} Oct 30 03:13:10.738190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.738099 4442 scope.go:122] "RemoveContainer" containerID="d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16" Oct 30 03:13:10.739471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.739450 4442 generic.go:356] "Generic (PLEG): container finished" podID="70f81e79-58a2-40e4-aa2a-464b18728733" containerID="84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9" exitCode=0 Oct 30 03:13:10.739548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.739473 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:10.739548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.739523 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" event={"ID":"70f81e79-58a2-40e4-aa2a-464b18728733","Type":"ContainerDied","Data":"84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9"} Oct 30 03:13:10.739616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.739546 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:10.739616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.739584 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:10.739856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.739842 4442 scope.go:122] "RemoveContainer" containerID="84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9" Oct 30 03:13:10.740990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.740964 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:10.741050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.741038 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:10.741098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.741083 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:13:10.817253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.817219 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 03:13:10.817397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.817280 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 03:13:10.843523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:10.843497 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:11.390780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.389952 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:11.746856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.746736 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" event={"ID":"70f81e79-58a2-40e4-aa2a-464b18728733","Type":"ContainerStarted","Data":"a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d"} Oct 30 03:13:11.749909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.749878 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" event={"ID":"107b2e13-5fb2-4241-8143-e75258d44261","Type":"ContainerStarted","Data":"81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791"} Oct 30 03:13:11.793779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.792186 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:13:11.793779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.792316 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:13:11.793779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.792360 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 03:13:11.809178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.809138 4442 patch_prober.go:29] interesting pod/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 03:13:11.809860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:11.809824 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 03:13:12.249030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.248991 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:12.298777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.298201 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:13:12.298777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.298344 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 03:13:12.298777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.298404 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 03:13:12.358265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.358235 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:12.797633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.797172 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:13:12.797633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.797247 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 03:13:12.827675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.826184 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:13:12.827675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.827524 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:13:12.835067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.834999 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 03:13:12.855857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:12.855809 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:13.006453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.006405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:13.152832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.151029 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:13.255769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.255715 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:13.281620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.281592 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 03:13:13.281789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.281654 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 03:13:13.406241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.406145 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:13:13.468105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.468064 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:13.790200 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.790163 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:13:13.790491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:13.790463 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 03:13:14.283182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:14.283143 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 03:13:14.283815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:14.283781 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 03:13:14.804780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:14.798860 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:13:14.804780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:14.798923 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 03:13:15.281843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:15.281806 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 03:13:15.282158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:15.282128 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 03:13:15.551545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:15.551454 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:13:15.636779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:15.636596 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:15.790789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:15.788165 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:13:15.790789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:15.788225 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 03:13:16.281313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:16.281280 4442 patch_prober.go:29] interesting pod/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 03:13:16.281557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:16.281529 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 03:13:16.557913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:16.557811 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:13:16.793779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:16.793149 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 03:13:16.793779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:16.793209 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 03:13:17.286081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:17.286051 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:13:17.287146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:17.287124 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 03:13:17.726426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:17.726343 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:17.796046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:17.795795 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:13:17.797308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:17.797247 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:13:19.132273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:19.132241 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:19.649067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:19.649035 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:20.123659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:20.123631 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:20.456677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:20.456598 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:20.766138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:20.766106 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:20.864801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:20.864746 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:20.970964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:20.970934 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:21.580037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:21.579995 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:23.336786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:23.336737 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:23.753158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:23.753127 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:13:24.861009 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:13:24.860975 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:13:25.249836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:25.249811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:27.025743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:27.025711 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:28.794724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:28.794686 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:30.646286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:30.646250 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:31.299908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:31.299875 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:32.237129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:32.237096 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:33.961967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:33.961934 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:13:34.039845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:34.039811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:34.713899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:34.713841 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:35.914068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:35.914032 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:37.630778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:37.630722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:38.154481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:38.154448 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:39.429545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:39.429507 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:41.119114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:41.119076 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:43.033168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:43.033132 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:43.233774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:43.233720 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:44.786611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:44.786570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:46.550112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:46.550080 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:48.227898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:48.227859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:49.094572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.094537 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc"] Oct 30 03:13:49.098430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.098410 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.124001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.123976 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"openshift-service-ca.crt\"" Oct 30 03:13:49.124168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.124039 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"kube-root-ca.crt\"" Oct 30 03:13:49.124168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.124060 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"premium-w4-8d80b1-kserve-self-signed-certs\"" Oct 30 03:13:49.124300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.124235 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d80b1\"/\"default-dockercfg-czx8v\"" Oct 30 03:13:49.126304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.126285 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc"] Oct 30 03:13:49.146450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.146423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:49.204062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.204035 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-dshm\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.204176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.204122 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5452058-93bf-4de6-a642-2dedbad09bd2-tls-certs\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.204283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.204264 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-model-cache\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.204358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.204342 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-tmp-dir\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.204524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.204510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxt66\" (UniqueName: \"kubernetes.io/projected/d5452058-93bf-4de6-a642-2dedbad09bd2-kube-api-access-xxt66\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.204559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.204535 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-home\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.306004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.305963 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5452058-93bf-4de6-a642-2dedbad09bd2-tls-certs\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.307103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.307076 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-model-cache\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.307322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.307305 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-tmp-dir\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.307511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.307489 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxt66\" (UniqueName: \"kubernetes.io/projected/d5452058-93bf-4de6-a642-2dedbad09bd2-kube-api-access-xxt66\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.309246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.307627 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-home\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.309603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.309580 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-dshm\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.312548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.309385 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-model-cache\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.315780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.309961 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-home\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.315915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.309122 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-tmp-dir\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.316047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.314858 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-dshm\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.316047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.314969 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5452058-93bf-4de6-a642-2dedbad09bd2-tls-certs\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.329534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.329507 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxt66\" (UniqueName: \"kubernetes.io/projected/d5452058-93bf-4de6-a642-2dedbad09bd2-kube-api-access-xxt66\") pod \"premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.347359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.347294 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:13:49.412744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.412714 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:49.714402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.714376 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc"] Oct 30 03:13:49.716801 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:13:49.716769 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5452058_93bf_4de6_a642_2dedbad09bd2.slice/crio-4bc5b8269ad3b8fa95dfbd2b3d37a4ca5ea6e3c4b83f9304c29c5f37819ef2b1 WatchSource:0}: Error finding container 4bc5b8269ad3b8fa95dfbd2b3d37a4ca5ea6e3c4b83f9304c29c5f37819ef2b1: Status 404 returned error can't find the container with id 4bc5b8269ad3b8fa95dfbd2b3d37a4ca5ea6e3c4b83f9304c29c5f37819ef2b1 Oct 30 03:13:49.929061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.929026 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" event={"ID":"d5452058-93bf-4de6-a642-2dedbad09bd2","Type":"ContainerStarted","Data":"9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4"} Oct 30 03:13:49.929061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.929063 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" event={"ID":"d5452058-93bf-4de6-a642-2dedbad09bd2","Type":"ContainerStarted","Data":"4bc5b8269ad3b8fa95dfbd2b3d37a4ca5ea6e3c4b83f9304c29c5f37819ef2b1"} Oct 30 03:13:49.967210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:49.967123 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" podStartSLOduration=0.967107359 podStartE2EDuration="967.107359ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:49.966966791 +0000 UTC m=+653.969319987" watchObservedRunningTime="2026-10-30 03:13:49.967107359 +0000 UTC m=+653.969460556" Oct 30 03:13:50.041940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:50.041913 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:50.412885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:50.412855 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:51.561859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:51.561819 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:51.735325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:51.735289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:53.526193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:53.526163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:54.874067 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:13:54.874024 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:13:55.228811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:55.228655 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:55.684530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:55.684491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:57.263422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:57.263390 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:59.028862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:59.028828 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:13:59.413584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:59.413497 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:59.426393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:59.426368 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:13:59.989492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:13:59.989467 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 03:14:00.729018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:00.728989 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:01.116378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:01.116346 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:14:02.927870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:02.927834 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:04.735913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:04.735877 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:06.447905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:06.447876 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:07.095649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:07.095612 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:08.316117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:08.316075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:10.008968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.008886 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:10.456239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.456148 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5"] Oct 30 03:14:10.461142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.461123 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.474582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.474560 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a9c9b6-openshift-default-dockercfg-9jv62\"" Oct 30 03:14:10.487797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.487772 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5"] Oct 30 03:14:10.536470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536439 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536470 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536607 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536714 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536673 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfbf4\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-kube-api-access-hfbf4\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536714 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536741 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bd0484-a942-4599-baa2-c8d1bb27e767-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536861 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536905 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.536981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.536963 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638082 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638051 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638087 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638151 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638171 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfbf4\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-kube-api-access-hfbf4\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638207 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638381 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bd0484-a942-4599-baa2-c8d1bb27e767-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638469 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638499 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638569 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638619 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638661 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.638872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.638849 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.639059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.639040 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bd0484-a942-4599-baa2-c8d1bb27e767-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.640768 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.640736 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.641041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.641024 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.661035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.661006 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.661035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.661030 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfbf4\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-kube-api-access-hfbf4\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:10.774409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:10.774386 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:11.089711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:11.089683 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5"] Oct 30 03:14:11.091061 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:14:11.091026 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bd0484_a942_4599_baa2_c8d1bb27e767.slice/crio-0f36608925416a28b03e38472874b2ec35cc6dfdeef86c4584388d28508bd7b1 WatchSource:0}: Error finding container 0f36608925416a28b03e38472874b2ec35cc6dfdeef86c4584388d28508bd7b1: Status 404 returned error can't find the container with id 0f36608925416a28b03e38472874b2ec35cc6dfdeef86c4584388d28508bd7b1 Oct 30 03:14:11.093234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:11.093207 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:14:11.093305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:11.093267 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:14:11.093305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:11.093295 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:14:11.856204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:11.856171 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:12.054092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:12.054061 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" event={"ID":"a8bd0484-a942-4599-baa2-c8d1bb27e767","Type":"ContainerStarted","Data":"7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d"} Oct 30 03:14:12.054254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:12.054111 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" event={"ID":"a8bd0484-a942-4599-baa2-c8d1bb27e767","Type":"ContainerStarted","Data":"0f36608925416a28b03e38472874b2ec35cc6dfdeef86c4584388d28508bd7b1"} Oct 30 03:14:12.119540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:12.119446 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" podStartSLOduration=2.119433168 podStartE2EDuration="2.119433168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:12.108195398 +0000 UTC m=+676.110548594" watchObservedRunningTime="2026-10-30 03:14:12.119433168 +0000 UTC m=+676.121786365" Oct 30 03:14:12.774853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:12.774825 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:12.776381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:12.776356 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:14:12.776501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:12.776401 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 03:14:13.775178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:13.775141 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:14:13.775689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:13.775305 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 03:14:13.818204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:13.818167 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:14.075093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:14.075004 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:14:14.774870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:14.774839 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:14:14.775048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:14.774890 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 03:14:15.775605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:15.775566 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:14:15.776123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:15.776085 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 03:14:15.850620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:15.850579 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:16.775420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:16.775380 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 03:14:16.775602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:16.775440 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 03:14:17.678375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:17.678334 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:17.778892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:17.778860 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:17.779140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:17.779120 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:17.779839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:17.779816 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 03:14:19.528664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:19.528633 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:21.335722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:21.335687 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:23.446955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:23.446915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:24.861850 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:14:24.861803 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:14:25.231931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:25.231852 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:27.025546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:27.025495 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:28.722427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:28.722388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:30.168165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:30.168134 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:14:30.662560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:30.662228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:32.514451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:32.514420 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:34.491632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:34.491597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:36.329274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:36.329232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:38.022137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:38.022105 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:39.813353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:39.813324 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:41.438315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:41.438283 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:43.130797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:43.130741 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:43.349589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:43.349550 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:45.021812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:45.021777 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:46.653417 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.653383 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt"] Oct 30 03:14:46.657009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.656987 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.683577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.683549 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e44313-openshift-default-dockercfg-tlmnm\"" Oct 30 03:14:46.686352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.686325 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt"] Oct 30 03:14:46.731702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.731661 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e14c69-96b5-4171-8684-64fa945d7d96-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.731702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.731706 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.731932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.731734 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlct6\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-kube-api-access-wlct6\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.731932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.731861 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.731932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.731888 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.731932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.731926 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.732064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.731989 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e14c69-96b5-4171-8684-64fa945d7d96-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.732064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.732030 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.732064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.732045 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.734533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.734513 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:46.833563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.833522 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e14c69-96b5-4171-8684-64fa945d7d96-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.833787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.833582 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.833787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.833627 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlct6\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-kube-api-access-wlct6\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.833787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.833660 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.833807 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.833889 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.833948 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e14c69-96b5-4171-8684-64fa945d7d96-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.834015 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.834040 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.834333 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e14c69-96b5-4171-8684-64fa945d7d96-istiod-ca-cert\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.834644 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-certs\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.834782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.834666 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-data\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.835023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.834815 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.835088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.835038 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-credential-socket\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.836790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.836746 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-envoy\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.837084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.837063 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e14c69-96b5-4171-8684-64fa945d7d96-istio-podinfo\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.860268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.860242 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlct6\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-kube-api-access-wlct6\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.862027 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.862004 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-istio-token\") pod \"e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:46.970534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:46.970469 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:47.316672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:47.316649 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt"] Oct 30 03:14:47.318428 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:14:47.318401 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e14c69_96b5_4171_8684_64fa945d7d96.slice/crio-f2be136df220b0f2daee80345d74f7fa46be0abeb4a1f18c6588e3831e4fc301 WatchSource:0}: Error finding container f2be136df220b0f2daee80345d74f7fa46be0abeb4a1f18c6588e3831e4fc301: Status 404 returned error can't find the container with id f2be136df220b0f2daee80345d74f7fa46be0abeb4a1f18c6588e3831e4fc301 Oct 30 03:14:47.320783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:47.320729 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:14:47.320881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:47.320817 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:14:47.320881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:47.320845 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:14:48.242366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:48.242325 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" event={"ID":"18e14c69-96b5-4171-8684-64fa945d7d96","Type":"ContainerStarted","Data":"ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477"} Oct 30 03:14:48.242782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:48.242391 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" event={"ID":"18e14c69-96b5-4171-8684-64fa945d7d96","Type":"ContainerStarted","Data":"f2be136df220b0f2daee80345d74f7fa46be0abeb4a1f18c6588e3831e4fc301"} Oct 30 03:14:48.306778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:48.302450 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" podStartSLOduration=2.302431223 podStartE2EDuration="2.302431223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:14:48.300623576 +0000 UTC m=+712.302976796" watchObservedRunningTime="2026-10-30 03:14:48.302431223 +0000 UTC m=+712.304784423" Oct 30 03:14:48.647496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:48.647464 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:48.971470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:48.971386 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:48.972871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:48.972846 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:14:48.972975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:48.972902 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 03:14:49.971494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:49.971460 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:14:49.971930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:49.971520 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 03:14:50.507982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:50.507944 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:50.713370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:50.713334 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:50.811549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:50.811457 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:50.970826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:50.970793 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:14:50.971005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:50.970843 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 03:14:51.970829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:51.970794 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:14:51.971326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:51.970854 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 03:14:52.319030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:52.318993 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:52.971650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:52.971614 4442 patch_prober.go:29] interesting pod/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 03:14:52.972091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:52.971663 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 03:14:53.985632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:53.985602 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:53.987069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:53.987038 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:53.989987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:53.989969 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:14:54.139143 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:54.139111 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:54.860434 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:14:54.860400 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:14:55.942043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:55.942007 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:57.726981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:57.726932 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:59.137651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:59.137609 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:14:59.605820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:14:59.605787 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:01.349001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:01.348970 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:03.226719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:03.226674 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:05.006961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:05.006932 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:05.885807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:05.885773 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:15:06.773519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:06.773484 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:08.486243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:08.486208 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:10.213117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:10.213083 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:11.922822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:11.922782 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:13.614203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:13.614147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:15.166735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.166702 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42"] Oct 30 03:15:15.170295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.170280 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.197093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.197061 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"default-dockercfg-48q8f\"" Oct 30 03:15:15.197244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.197097 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"kube-root-ca.crt\"" Oct 30 03:15:15.197244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.197188 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"distinct-w0-a9c9b6-kserve-self-signed-certs\"" Oct 30 03:15:15.197332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.197287 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a9c9b6\"/\"openshift-service-ca.crt\"" Oct 30 03:15:15.199319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.199298 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42"] Oct 30 03:15:15.220270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.220243 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:15.260884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.260858 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-model-cache\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.261000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.260890 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-home\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.261000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.260915 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-dshm\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.261073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.261020 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tls-certs\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.261152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.261136 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vk94\" (UniqueName: \"kubernetes.io/projected/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-kube-api-access-9vk94\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.261187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.261173 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tmp-dir\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.373578 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.373512 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vk94\" (UniqueName: \"kubernetes.io/projected/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-kube-api-access-9vk94\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.374467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.374420 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tmp-dir\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.374782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.374741 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-model-cache\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.374972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.374956 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-home\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.375150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.375135 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-dshm\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.375354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.375315 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tls-certs\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.395780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.375573 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-home\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.395780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.375090 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tmp-dir\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.396840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.375361 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-model-cache\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.411325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.406928 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-dshm\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.411325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.409396 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tls-certs\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.411732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.411701 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:15.414057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.413835 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vk94\" (UniqueName: \"kubernetes.io/projected/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-kube-api-access-9vk94\") pod \"distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.488120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.487981 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:15.790850 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:15.790823 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42"] Oct 30 03:15:15.792479 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:15:15.792451 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e568b64_2cef_4f47_9ff6_fb8b4c576cf1.slice/crio-105e73945fbd2450afe79d52a3605931f95c104a7f5be3d97c66d2424fdf5e88 WatchSource:0}: Error finding container 105e73945fbd2450afe79d52a3605931f95c104a7f5be3d97c66d2424fdf5e88: Status 404 returned error can't find the container with id 105e73945fbd2450afe79d52a3605931f95c104a7f5be3d97c66d2424fdf5e88 Oct 30 03:15:16.420226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:16.420193 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" event={"ID":"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1","Type":"ContainerStarted","Data":"6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69"} Oct 30 03:15:16.420602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:16.420235 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" event={"ID":"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1","Type":"ContainerStarted","Data":"105e73945fbd2450afe79d52a3605931f95c104a7f5be3d97c66d2424fdf5e88"} Oct 30 03:15:16.461466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:16.461419 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" podStartSLOduration=1.461407452 podStartE2EDuration="1.461407452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:16.459429387 +0000 UTC m=+740.461782584" watchObservedRunningTime="2026-10-30 03:15:16.461407452 +0000 UTC m=+740.463760649" Oct 30 03:15:16.489913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:16.489880 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:17.123383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:17.123352 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:18.829605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:18.829570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:19.576062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:19.576027 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:20.608358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:20.608314 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:22.230633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:22.230599 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:22.782475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:22.782439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:23.921041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:23.921005 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:24.860162 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:15:24.860127 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:15:25.489435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:25.489405 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:25.502291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:25.502268 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:25.614807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:25.614777 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:26.478249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:26.478220 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 03:15:27.342433 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:27.342387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:28.644411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:28.644371 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:15:29.118635 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:29.118603 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:30.949932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:30.949897 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:32.612521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:32.612484 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:34.342602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:34.342567 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:35.936959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:35.936931 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:36.507659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:36.507631 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:37.611440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:37.611406 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:39.336129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:39.336049 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:40.732269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.732240 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql"] Oct 30 03:15:40.737059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.737037 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.758798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.758743 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.758944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.758854 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.759032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.758947 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d818414-c583-416a-9561-927249e337e5-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.759032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.759013 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d818414-c583-416a-9561-927249e337e5-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.759176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.759053 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.759176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.759138 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.759258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.759207 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d76s\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-kube-api-access-4d76s\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.759293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.759270 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.759345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.759297 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.765662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.765638 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql"] Oct 30 03:15:40.860827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.860792 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d818414-c583-416a-9561-927249e337e5-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.860995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.860833 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d818414-c583-416a-9561-927249e337e5-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.860995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.860951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861003 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861040 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d76s\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-kube-api-access-4d76s\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861083 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861103 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861193 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861224 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861371 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861395 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861512 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861657 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.861942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.861806 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d818414-c583-416a-9561-927249e337e5-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.863477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.863453 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.863565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.863557 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d818414-c583-416a-9561-927249e337e5-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.878527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.878500 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:40.878634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:40.878620 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d76s\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-kube-api-access-4d76s\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:41.021009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.020983 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:41.050745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.050715 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:41.387612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.387477 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql"] Oct 30 03:15:41.390403 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.390363 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:15:41.390537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.390491 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:15:41.390595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.390569 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:15:41.554960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.554917 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" event={"ID":"9d818414-c583-416a-9561-927249e337e5","Type":"ContainerStarted","Data":"ee5a0b546004cae491639ecf24fe0b9ef4000a06802b4f6dd0ff7d9e1202aaa1"} Oct 30 03:15:41.555149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.554985 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" event={"ID":"9d818414-c583-416a-9561-927249e337e5","Type":"ContainerStarted","Data":"5ae0f8dc03764197831dc7d457c12dfd9a6074c9cae5385a82c122f5aebd6895"} Oct 30 03:15:41.598317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:41.598253 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podStartSLOduration=1.598235273 podStartE2EDuration="1.598235273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:15:41.596324543 +0000 UTC m=+765.598677731" watchObservedRunningTime="2026-10-30 03:15:41.598235273 +0000 UTC m=+765.600588471" Oct 30 03:15:42.051387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:42.051348 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:42.053131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:42.053090 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:15:42.053282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:42.053141 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 03:15:43.030989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:43.030959 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:43.051000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:43.050965 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:15:43.051160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:43.051019 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 03:15:43.494419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:43.494303 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:44.051304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:44.051270 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:15:44.051479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:44.051327 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 03:15:44.829871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:44.829814 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:45.051534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:45.051502 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:15:45.051717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:45.051554 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 03:15:45.259100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:45.259071 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:46.051430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:46.051395 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:15:46.051950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:46.051449 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 03:15:46.648395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:46.648362 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:46.809079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:46.809045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:47.051825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:47.051788 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 03:15:47.052234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:47.051841 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 03:15:48.345844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:48.345816 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:48.346339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:48.346101 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:48.347398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:48.347375 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:15:48.631559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:48.631467 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:50.213736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:50.213704 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:51.971919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:51.971887 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:53.704858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:53.704823 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:54.235314 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:54.235278 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:15:54.894150 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:15:54.894115 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:15:55.415474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:55.415439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:57.105137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:57.105100 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:15:58.727055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:15:58.727009 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:00.410203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:00.410168 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:01.943061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:01.943023 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl"] Oct 30 03:16:01.947192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:01.947172 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:01.968787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:01.964104 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"openshift-service-ca.crt\"" Oct 30 03:16:01.968787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:01.964195 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"kube-root-ca.crt\"" Oct 30 03:16:01.975563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:01.975076 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"default-dockercfg-cqsbs\"" Oct 30 03:16:01.976132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:01.976092 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-e44313\"/\"distinct-w1-e44313-kserve-self-signed-certs\"" Oct 30 03:16:01.979045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:01.979020 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl"] Oct 30 03:16:02.017275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.017238 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:02.031186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.031157 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-model-cache\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.031386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.031366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b752eca-931a-4222-8569-8bae91568839-tls-certs\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.031459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.031401 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fht67\" (UniqueName: \"kubernetes.io/projected/7b752eca-931a-4222-8569-8bae91568839-kube-api-access-fht67\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.031459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.031444 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-home\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.031564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.031468 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-dshm\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.031564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.031501 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-tmp-dir\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133049 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b752eca-931a-4222-8569-8bae91568839-tls-certs\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133088 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fht67\" (UniqueName: \"kubernetes.io/projected/7b752eca-931a-4222-8569-8bae91568839-kube-api-access-fht67\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133113 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-home\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133130 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-dshm\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133151 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-tmp-dir\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133262 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-model-cache\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133558 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-home\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.133727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.133703 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-tmp-dir\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.134040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.134018 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-model-cache\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.135559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.135541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-dshm\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.135721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.135706 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b752eca-931a-4222-8569-8bae91568839-tls-certs\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.138336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.138307 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:02.152221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.152196 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fht67\" (UniqueName: \"kubernetes.io/projected/7b752eca-931a-4222-8569-8bae91568839-kube-api-access-fht67\") pod \"distinct-w1-e44313-kserve-667854f86d-sz2vl\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.260654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.260615 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:02.562928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.562847 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl"] Oct 30 03:16:02.570285 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:16:02.570260 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b752eca_931a_4222_8569_8bae91568839.slice/crio-3a02434c2863553b1c2858d5afaf803ea8430fdae40a1a8c571882ea46c0aa7c WatchSource:0}: Error finding container 3a02434c2863553b1c2858d5afaf803ea8430fdae40a1a8c571882ea46c0aa7c: Status 404 returned error can't find the container with id 3a02434c2863553b1c2858d5afaf803ea8430fdae40a1a8c571882ea46c0aa7c Oct 30 03:16:02.665426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.665389 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" event={"ID":"7b752eca-931a-4222-8569-8bae91568839","Type":"ContainerStarted","Data":"fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9"} Oct 30 03:16:02.665579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.665434 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" event={"ID":"7b752eca-931a-4222-8569-8bae91568839","Type":"ContainerStarted","Data":"3a02434c2863553b1c2858d5afaf803ea8430fdae40a1a8c571882ea46c0aa7c"} Oct 30 03:16:02.704474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:02.704427 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" podStartSLOduration=1.704412247 podStartE2EDuration="1.704412247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:02.702910721 +0000 UTC m=+786.705263919" watchObservedRunningTime="2026-10-30 03:16:02.704412247 +0000 UTC m=+786.706765445" Oct 30 03:16:03.261228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:03.261193 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:03.920506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:03.920470 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:05.714984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:05.709677 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:07.405107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:07.405071 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:08.378934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:08.378901 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:16:09.131481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:09.131420 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:10.914996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:10.914965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:11.948469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:11.948434 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:12.048434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:12.048408 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:12.261381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:12.261350 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:12.274294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:12.274270 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:12.609887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:12.609791 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:12.728857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:12.728832 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 03:16:14.323121 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:14.323087 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:16.033135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:16.033101 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:17.815870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:17.815835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:19.530636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:19.530602 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:21.318991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:21.318959 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:23.025740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:23.025703 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:24.806730 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:24.806680 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:24.859790 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:16:24.859735 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:16:25.352571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:25.352541 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:25.443255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:25.443220 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:26.507723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:26.507692 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:28.106462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:28.106423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:29.798068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:29.798031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:31.512806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:31.512773 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:33.130018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:33.129987 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:34.808592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:34.808553 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:36.506455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:36.506421 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:38.125941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:38.125903 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:39.780789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:39.780729 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:41.411057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:41.411022 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:43.080885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:43.080850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:43.608584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:43.608538 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:44.792318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:44.792288 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:46.234347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:46.234305 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:46.534813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:46.534780 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:48.200712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:48.200671 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw"] Oct 30 03:16:48.201168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:48.200892 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" containerID="cri-o://156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827" gracePeriod=30 Oct 30 03:16:48.253466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:48.253432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:50.008916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:50.008877 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:51.706244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:51.706202 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:52.708102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:52.708065 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:16:53.315017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.314990 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:53.528054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.528031 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:16:53.614786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.614736 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-podinfo\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.614945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.614796 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-certs\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.614945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.614923 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-data\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.615079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615048 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-kube-api-access-b4pzz\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-kube-api-access-b4pzz\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.615143 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615118 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-token\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-token\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.615207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615143 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-certs" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:53.615265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615208 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-socket\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.615265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615238 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-envoy\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.615265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615238 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-data" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:53.615425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615278 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istiod-ca-cert\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.615425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615314 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-credential-socket\") pod \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\" (UID: \"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5\") " Oct 30 03:16:53.615957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615678 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-socket" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:53.615957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615925 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.615957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615947 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.615957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.615960 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.616463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.616271 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istiod-ca-cert" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:16:53.616617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.616594 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-credential-socket" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:53.618024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.617989 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-kube-api-access-b4pzz" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "kube-api-access-b4pzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:16:53.618024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.617996 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-podinfo" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:16:53.618331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.618306 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-envoy" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:16:53.618836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.618817 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-token" pod "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" (UID: "5871f77f-4012-4c6e-a0ef-ffb5814bcdc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:16:53.717254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.717224 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.717254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.717248 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.717254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.717257 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.717704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.717266 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.717704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.717275 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.717704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.717283 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pzz\" (UniqueName: \"kubernetes.io/projected/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5-kube-api-access-b4pzz\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:16:53.945954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.945867 4442 generic.go:356] "Generic (PLEG): container finished" podID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerID="156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827" exitCode=0 Oct 30 03:16:53.945954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.945928 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" event={"ID":"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5","Type":"ContainerDied","Data":"156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827"} Oct 30 03:16:53.946184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.945976 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" event={"ID":"5871f77f-4012-4c6e-a0ef-ffb5814bcdc5","Type":"ContainerDied","Data":"3f5027478ac1388bf7bc3c7c7bfb1dde516d92a021f08a2d822e3a26201488e1"} Oct 30 03:16:53.946184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.945932 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw" Oct 30 03:16:53.946184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.946003 4442 scope.go:122] "RemoveContainer" containerID="156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827" Oct 30 03:16:53.956147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.956105 4442 scope.go:122] "RemoveContainer" containerID="156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827" Oct 30 03:16:53.956624 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:16:53.956562 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827\": container with ID starting with 156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827 not found: ID does not exist" containerID="156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827" Oct 30 03:16:53.956624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.956599 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827"} err="failed to get container status \"156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827\": rpc error: code = NotFound desc = could not find container \"156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827\": container with ID starting with 156e84f18eba3460ab1c5d9b7c5b7b97998970749b75e694aec3c9af49c1f827 not found: ID does not exist" Oct 30 03:16:53.995421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:53.995391 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw"] Oct 30 03:16:54.005658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:54.005623 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-qb9nw"] Oct 30 03:16:54.109472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:54.109434 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" path="/var/lib/kubelet/pods/5871f77f-4012-4c6e-a0ef-ffb5814bcdc5/volumes" Oct 30 03:16:54.860625 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:16:54.860583 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:16:55.039013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:55.038948 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:56.814438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:56.814404 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:16:58.517271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:16:58.517239 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:00.307609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:00.307571 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:02.023269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:02.023172 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:03.682394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:03.682358 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:05.308879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:05.308845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:06.998541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:06.998507 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:08.610187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:08.610152 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:10.414647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:10.414566 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:12.014550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:12.014517 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:13.620773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:13.620729 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:15.285901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:15.285867 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:16.900863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:16.900826 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:18.588720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:18.588685 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:20.310052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:20.310019 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:21.991743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:21.991708 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:23.581434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:23.581405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:24.860509 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:17:24.860467 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:17:25.241562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:25.241487 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:26.919268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:26.919229 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:28.616719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:28.616689 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:30.324933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:30.324905 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:30.757929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:30.757900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:31.912511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:31.912481 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:33.615345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:33.615310 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:35.211862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:35.211830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:36.820917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:36.820885 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:38.397210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:38.397177 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:40.105456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:40.105424 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:41.703141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:41.703111 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:43.306697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:43.306666 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:43.714149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:43.714069 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:44.905239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:44.905207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:46.566360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:46.566328 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:48.302531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:48.302502 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:49.904490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:49.904451 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:51.498569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:51.498537 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:53.171377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:53.171349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:54.746967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:54.746929 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:54.859498 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:17:54.859459 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:17:56.192081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:56.192054 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:17:56.193627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:56.193603 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:17:56.341381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:56.341333 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:57.937912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:57.937878 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:17:59.632738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:17:59.632700 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:01.326834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:01.326801 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:02.926040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:02.925998 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:04.531304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:04.531274 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:06.224193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:06.224160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:07.812879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:07.812843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:09.412227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:09.412167 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:11.007225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:11.007190 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:12.699769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:12.699711 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:14.206074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:14.206027 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:15.805964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:15.805929 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:17.411113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:17.411081 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:19.019268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:19.019198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:20.641251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:20.641213 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:22.266729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:22.266698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:23.885367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:23.885334 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:24.861233 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:18:24.861199 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:18:25.496627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:25.496596 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:27.077864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:27.077799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:28.676121 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:28.676094 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:30.309972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:30.309939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:31.935053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:31.935021 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:33.529471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:33.529432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:35.223691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:35.223655 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:36.915692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:36.915653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:38.608360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:38.608327 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:40.333157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:40.333079 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:42.011093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:42.011057 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:42.113178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:42.113133 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:42.487071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:42.486994 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:42.619512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:42.619475 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:43.724349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:43.724308 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:43.824715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:43.824680 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:45.019867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:45.019833 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:45.124653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:45.124606 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:45.224309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:45.224279 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:45.537910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:45.537879 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:45.933259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:45.933180 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:46.033385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:46.033345 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:46.367095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:46.367063 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:46.469842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:46.469805 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:46.566851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:46.566814 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:47.615363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.615330 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:47.718614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.718587 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:47.785474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.785445 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg"] Oct 30 03:18:47.785940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.785912 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" Oct 30 03:18:47.785940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.785929 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" Oct 30 03:18:47.786127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.785996 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5871f77f-4012-4c6e-a0ef-ffb5814bcdc5" containerName="istio-proxy" Oct 30 03:18:47.789160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.789140 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:47.807138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.807117 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg"] Oct 30 03:18:47.833868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.833835 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b132360f\"" Oct 30 03:18:47.865114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.865081 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b132360f-dockercfg-xpjjr\"" Oct 30 03:18:47.920916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.920820 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffwd\" (UniqueName: \"kubernetes.io/projected/c7cd6588-486b-411a-8365-fc57c97da59f-kube-api-access-nffwd\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:47.921228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:47.921204 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cd6588-486b-411a-8365-fc57c97da59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:48.022007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.021974 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cd6588-486b-411a-8365-fc57c97da59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:48.022199 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.022118 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffwd\" (UniqueName: \"kubernetes.io/projected/c7cd6588-486b-411a-8365-fc57c97da59f-kube-api-access-nffwd\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:48.022613 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.022595 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cd6588-486b-411a-8365-fc57c97da59f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:48.040028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.040000 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffwd\" (UniqueName: \"kubernetes.io/projected/c7cd6588-486b-411a-8365-fc57c97da59f-kube-api-access-nffwd\") pod \"payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:48.102861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.102830 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:48.399859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.399835 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg"] Oct 30 03:18:48.401714 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:18:48.401688 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cd6588_486b_411a_8365_fc57c97da59f.slice/crio-52442c5d4c6125092923c42a807f7ebf4e7c7b422f518bcf206dc1c936c7ffe9 WatchSource:0}: Error finding container 52442c5d4c6125092923c42a807f7ebf4e7c7b422f518bcf206dc1c936c7ffe9: Status 404 returned error can't find the container with id 52442c5d4c6125092923c42a807f7ebf4e7c7b422f518bcf206dc1c936c7ffe9 Oct 30 03:18:48.558104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.558068 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" event={"ID":"c7cd6588-486b-411a-8365-fc57c97da59f","Type":"ContainerStarted","Data":"fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36"} Oct 30 03:18:48.558272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.558146 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" event={"ID":"c7cd6588-486b-411a-8365-fc57c97da59f","Type":"ContainerStarted","Data":"52442c5d4c6125092923c42a807f7ebf4e7c7b422f518bcf206dc1c936c7ffe9"} Oct 30 03:18:48.558272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.558176 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:48.594708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:48.594655 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" podStartSLOduration=1.594638868 podStartE2EDuration="1.594638868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:48.592217417 +0000 UTC m=+952.594570614" watchObservedRunningTime="2026-10-30 03:18:48.594638868 +0000 UTC m=+952.596992064" Oct 30 03:18:49.414326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:49.414293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:51.053192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:51.053160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:52.820098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:52.820065 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:54.627324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:54.627283 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:54.859832 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:18:54.859804 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:18:56.216573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.216538 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss"] Oct 30 03:18:56.220192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.220176 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.229599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.229572 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss"] Oct 30 03:18:56.236830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.236804 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b132360f-serving-cert\"" Oct 30 03:18:56.298583 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.298543 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:56.317817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.317777 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da698ab-d12f-4add-8361-3a2ddf970672-openshift-service-ca\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.317992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.317917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-metrics-certs\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.317992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.317942 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xq2\" (UniqueName: \"kubernetes.io/projected/2da698ab-d12f-4add-8361-3a2ddf970672-kube-api-access-76xq2\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.318073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.318047 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-maas-api-tls\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.419023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.418945 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:56.425786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.421397 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da698ab-d12f-4add-8361-3a2ddf970672-openshift-service-ca\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.428317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.428274 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-metrics-certs\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.428556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.428540 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xq2\" (UniqueName: \"kubernetes.io/projected/2da698ab-d12f-4add-8361-3a2ddf970672-kube-api-access-76xq2\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.428880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.428842 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-maas-api-tls\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.429179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.422616 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da698ab-d12f-4add-8361-3a2ddf970672-openshift-service-ca\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.432983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.432959 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-maas-api-tls\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.433090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.433035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-metrics-certs\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.447152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.447123 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xq2\" (UniqueName: \"kubernetes.io/projected/2da698ab-d12f-4add-8361-3a2ddf970672-kube-api-access-76xq2\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.500415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.500386 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:56.534860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.534833 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:56.830938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.830911 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss"] Oct 30 03:18:56.832543 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:18:56.832518 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da698ab_d12f_4add_8361_3a2ddf970672.slice/crio-b61628494c9c5b3e3f1dfccdbab75f19a90c847a6098a07937f76514ab483e56 WatchSource:0}: Error finding container b61628494c9c5b3e3f1dfccdbab75f19a90c847a6098a07937f76514ab483e56: Status 404 returned error can't find the container with id b61628494c9c5b3e3f1dfccdbab75f19a90c847a6098a07937f76514ab483e56 Oct 30 03:18:56.833955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:56.833937 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:18:57.611294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:57.611205 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" event={"ID":"2da698ab-d12f-4add-8361-3a2ddf970672","Type":"ContainerStarted","Data":"90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2"} Oct 30 03:18:57.611294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:57.611262 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" event={"ID":"2da698ab-d12f-4add-8361-3a2ddf970672","Type":"ContainerStarted","Data":"b61628494c9c5b3e3f1dfccdbab75f19a90c847a6098a07937f76514ab483e56"} Oct 30 03:18:57.611704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:57.611306 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:18:57.649799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:57.649737 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" podStartSLOduration=1.214736469 podStartE2EDuration="1.6497221s" totalImagesPullingTime="434.985631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:18:57.647970683 +0000 UTC m=+961.650323882" watchObservedRunningTime="2026-10-30 03:18:57.6497221 +0000 UTC m=+961.652075297" Oct 30 03:18:58.127918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:58.127871 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:58.389404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:58.389315 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:18:59.567033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:59.567003 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:18:59.753981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:18:59.753950 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:01.423532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:01.423498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:03.030173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:03.030142 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:03.625198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:03.625169 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:19:04.778996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:04.778965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:04.899921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:04.899884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:06.198318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:06.198289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:06.520955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:06.520915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:06.701721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:06.701683 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:07.001015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:07.000983 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:08.229775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:08.229720 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:08.958810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:08.958777 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:19:10.075076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:10.075043 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:10.188902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:10.188861 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:10.287323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:10.287287 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:11.039302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:11.039266 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:11.855913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:11.855883 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:11.953381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:11.953345 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:12.027073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.027046 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf"] Oct 30 03:19:12.031039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.031023 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.048619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.048594 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf"] Oct 30 03:19:12.051618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.051593 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:12.057135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.057105 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f247d1\"" Oct 30 03:19:12.057135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.057118 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f247d1-dockercfg-fgcln\"" Oct 30 03:19:12.233746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.233667 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wst65\" (UniqueName: \"kubernetes.io/projected/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-kube-api-access-wst65\") pod \"payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.233914 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.233821 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.334727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.334680 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wst65\" (UniqueName: \"kubernetes.io/projected/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-kube-api-access-wst65\") pod \"payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.334979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.334815 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.335522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.335495 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.355341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.355304 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wst65\" (UniqueName: \"kubernetes.io/projected/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-kube-api-access-wst65\") pod \"payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.647038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.646986 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:12.957697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:12.957663 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf"] Oct 30 03:19:12.958157 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:19:12.958130 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5cd4573_0ed7_40a3_bb2a_c01060e9021a.slice/crio-753b3e9c705a9b4133d950f94139eb6e904407f961cbcdd7c320c7e2f820640d WatchSource:0}: Error finding container 753b3e9c705a9b4133d950f94139eb6e904407f961cbcdd7c320c7e2f820640d: Status 404 returned error can't find the container with id 753b3e9c705a9b4133d950f94139eb6e904407f961cbcdd7c320c7e2f820640d Oct 30 03:19:13.067390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:13.067364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:13.699581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:13.699545 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" event={"ID":"e5cd4573-0ed7-40a3-bb2a-c01060e9021a","Type":"ContainerStarted","Data":"15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11"} Oct 30 03:19:13.699794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:13.699602 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" event={"ID":"e5cd4573-0ed7-40a3-bb2a-c01060e9021a","Type":"ContainerStarted","Data":"753b3e9c705a9b4133d950f94139eb6e904407f961cbcdd7c320c7e2f820640d"} Oct 30 03:19:13.699794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:13.699781 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:13.736200 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:13.736151 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" podStartSLOduration=1.7361375319999999 podStartE2EDuration="1.736137532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:13.734775589 +0000 UTC m=+977.737128807" watchObservedRunningTime="2026-10-30 03:19:13.736137532 +0000 UTC m=+977.738490729" Oct 30 03:19:13.810208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:13.810174 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:15.474426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:15.474395 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:17.139996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:17.139957 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:17.930364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:17.930334 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:18.347204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.347166 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq"] Oct 30 03:19:18.351312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.351266 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.364581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.364559 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-585938-openshift-default-dockercfg-nzl86\"" Oct 30 03:19:18.377896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.377868 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq"] Oct 30 03:19:18.398245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.398219 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:18.414770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.414731 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-socket\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.414889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.414841 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-envoy\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.414934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.414893 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-data\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.414934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.414917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-podinfo\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.415041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.414934 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-token\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.415041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.414960 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-certs\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.415116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.415079 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istiod-ca-cert\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.415165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.415119 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-credential-socket\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.415165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.415137 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbdl\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-kube-api-access-7xbdl\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.516064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.516026 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-envoy\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.516337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.516316 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-data\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.516476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.516459 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-podinfo\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.516587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.516573 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-token\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.516709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.516695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-certs\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.516865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.516849 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istiod-ca-cert\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.516997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.516978 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-credential-socket\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.517115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.517099 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbdl\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-kube-api-access-7xbdl\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.520563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.517290 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-socket\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.521258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.521234 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-socket\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.522018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.521994 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-certs\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.524838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.522932 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-credential-socket\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.524995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.524234 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-data\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.525122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.524499 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istiod-ca-cert\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.542405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.542305 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-podinfo\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.554676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.554643 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbdl\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-kube-api-access-7xbdl\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.555133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.555017 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-envoy\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.557206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.557181 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-token\") pod \"e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.631881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.631803 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:18.652778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.652727 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr"] Oct 30 03:19:18.657659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.657640 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.670318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.670294 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:18.707361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.707331 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr"] Oct 30 03:19:18.722381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722352 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722424 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722584 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsf8q\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-kube-api-access-vsf8q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722628 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722685 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722840 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722908 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722938 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.722974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.722967 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824257 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824289 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824308 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824354 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824548 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsf8q\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-kube-api-access-vsf8q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824607 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824651 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.824704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824671 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.825001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824724 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.825001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824732 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.825001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.824835 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.825156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.825039 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.825156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.825105 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.827088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.827070 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.827227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.827209 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.844301 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.844271 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsf8q\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-kube-api-access-vsf8q\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.844411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.844308 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-k6nmr\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:18.850705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:18.850683 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:19.000839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.000814 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:19.035341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.035303 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq"] Oct 30 03:19:19.044844 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:19:19.044808 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3313fb_3e34_42bc_a4b5_aa52df7756ec.slice/crio-0f517bbbae60b12c3a7e23dd9b72b506f4d963843a33ef72a6e871bf4185173e WatchSource:0}: Error finding container 0f517bbbae60b12c3a7e23dd9b72b506f4d963843a33ef72a6e871bf4185173e: Status 404 returned error can't find the container with id 0f517bbbae60b12c3a7e23dd9b72b506f4d963843a33ef72a6e871bf4185173e Oct 30 03:19:19.047990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.047952 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:19.048232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.048206 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:19.048329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.048265 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:19.335504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.335479 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr"] Oct 30 03:19:19.337095 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:19:19.337055 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f928cd_d051_43f2_a6d2_6d40bdc0c635.slice/crio-d18de72a658aa7c23e24e4b9d75dfdb778ff8bc9a624b2051eb8f2adfd246eec WatchSource:0}: Error finding container d18de72a658aa7c23e24e4b9d75dfdb778ff8bc9a624b2051eb8f2adfd246eec: Status 404 returned error can't find the container with id d18de72a658aa7c23e24e4b9d75dfdb778ff8bc9a624b2051eb8f2adfd246eec Oct 30 03:19:19.340120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.340089 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:19.340218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.340196 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:19.340280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.340243 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:19.735661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.735579 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" event={"ID":"9e3313fb-3e34-42bc-a4b5-aa52df7756ec","Type":"ContainerStarted","Data":"3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e"} Oct 30 03:19:19.736116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.735899 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" event={"ID":"9e3313fb-3e34-42bc-a4b5-aa52df7756ec","Type":"ContainerStarted","Data":"0f517bbbae60b12c3a7e23dd9b72b506f4d963843a33ef72a6e871bf4185173e"} Oct 30 03:19:19.737528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.737503 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" event={"ID":"10f928cd-d051-43f2-a6d2-6d40bdc0c635","Type":"ContainerStarted","Data":"53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace"} Oct 30 03:19:19.737653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.737540 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" event={"ID":"10f928cd-d051-43f2-a6d2-6d40bdc0c635","Type":"ContainerStarted","Data":"d18de72a658aa7c23e24e4b9d75dfdb778ff8bc9a624b2051eb8f2adfd246eec"} Oct 30 03:19:19.776658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.776612 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" podStartSLOduration=1.776597389 podStartE2EDuration="1.776597389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:19.775194395 +0000 UTC m=+983.777547591" watchObservedRunningTime="2026-10-30 03:19:19.776597389 +0000 UTC m=+983.778950588" Oct 30 03:19:19.817152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:19.817100 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podStartSLOduration=1.817087381 podStartE2EDuration="1.817087381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:19.813889471 +0000 UTC m=+983.816242665" watchObservedRunningTime="2026-10-30 03:19:19.817087381 +0000 UTC m=+983.819440618" Oct 30 03:19:20.001931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.001908 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:20.003501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.003473 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:20.003659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.003557 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:20.467090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.467002 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:20.564983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.564952 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:20.672517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.672485 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:20.678292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.678265 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:20.699779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.696045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:20.742568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.742470 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:20.743396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:20.743379 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:21.012777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:21.004073 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:21.012777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:21.004139 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:22.009801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.004776 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:22.009801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.004841 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:22.180168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.180139 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:22.286839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.282897 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:22.513506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.513462 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm"] Oct 30 03:19:22.518256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.518234 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.541619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.541543 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm"] Oct 30 03:19:22.547969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.547942 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f247d1-serving-cert\"" Oct 30 03:19:22.579158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.579131 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-metrics-certs\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.579301 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.579185 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-maas-api-tls\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.579301 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.579217 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-openshift-service-ca\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.579476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.579451 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhnzv\" (UniqueName: \"kubernetes.io/projected/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-kube-api-access-bhnzv\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.615739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.615709 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:22.681045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.681015 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-openshift-service-ca\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.681222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.681134 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhnzv\" (UniqueName: \"kubernetes.io/projected/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-kube-api-access-bhnzv\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.681222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.681202 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-metrics-certs\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.681340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.681258 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-maas-api-tls\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.681710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.681684 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-openshift-service-ca\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.710386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.710345 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-maas-api-tls\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.711000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.710972 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-metrics-certs\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.712251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.712222 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhnzv\" (UniqueName: \"kubernetes.io/projected/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-kube-api-access-bhnzv\") pod \"maas-api-e2e-api-a-f247d1-794db7449d-dbbjm\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:22.715360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.715338 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:22.842609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:22.842527 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:23.012935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.007702 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:23.012935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.007797 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:23.168029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.167995 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm"] Oct 30 03:19:23.169213 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:19:23.169182 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359cbe0c_3723_4266_8fa5_fc5b2a0a63da.slice/crio-42696cdee67ab229fa7e7176620c1265cce27701e19d072e92acfa96c322fcd3 WatchSource:0}: Error finding container 42696cdee67ab229fa7e7176620c1265cce27701e19d072e92acfa96c322fcd3: Status 404 returned error can't find the container with id 42696cdee67ab229fa7e7176620c1265cce27701e19d072e92acfa96c322fcd3 Oct 30 03:19:23.829740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.829644 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" event={"ID":"359cbe0c-3723-4266-8fa5-fc5b2a0a63da","Type":"ContainerStarted","Data":"b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8"} Oct 30 03:19:23.829740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.829698 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" event={"ID":"359cbe0c-3723-4266-8fa5-fc5b2a0a63da","Type":"ContainerStarted","Data":"42696cdee67ab229fa7e7176620c1265cce27701e19d072e92acfa96c322fcd3"} Oct 30 03:19:23.829740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.829735 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:23.869134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.869088 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" podStartSLOduration=1.477727265 podStartE2EDuration="1.8690745s" totalImagesPullingTime="391.347235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:23.868362357 +0000 UTC m=+987.870715555" watchObservedRunningTime="2026-10-30 03:19:23.8690745 +0000 UTC m=+987.871427719" Oct 30 03:19:23.938972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:23.938939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:24.006338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:24.005717 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:24.006338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:24.005806 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:24.049464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:24.049432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:24.661045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:24.661014 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:24.739047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:24.734191 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:19:24.862114 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:19:24.862078 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:19:25.006777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:25.003074 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:25.006777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:25.003133 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:26.001694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:26.001659 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:26.002091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:26.001720 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:26.435491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:26.435399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:27.001246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:27.001210 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 03:19:27.001413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:27.001278 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 03:19:28.038272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:28.038241 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:28.038799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:28.038580 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:28.039654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:28.039632 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:19:28.226065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:28.226032 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:29.842222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:29.842194 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:19:29.915008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:29.914977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:31.522363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:31.522330 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:33.316469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:33.316435 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:35.114684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:35.114653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:36.599112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.599074 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m"] Oct 30 03:19:36.603388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.603367 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.616957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.616880 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-585938-serving-cert\"" Oct 30 03:19:36.630591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.630566 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m"] Oct 30 03:19:36.679189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.679159 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:36.779811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.779741 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:36.784111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.784085 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-maas-api-tls\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.784260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.784234 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-openshift-service-ca\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.784369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.784349 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f57vq\" (UniqueName: \"kubernetes.io/projected/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-kube-api-access-f57vq\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.784432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.784408 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-metrics-certs\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.885734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.885646 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-metrics-certs\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.885734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.885726 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-maas-api-tls\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.885933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.885842 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-openshift-service-ca\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.885933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.885897 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f57vq\" (UniqueName: \"kubernetes.io/projected/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-kube-api-access-f57vq\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.886514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.886487 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-openshift-service-ca\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.888297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.888266 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-maas-api-tls\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.888388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.888372 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-metrics-certs\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.891733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.891714 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:19:36.903137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.903116 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f57vq\" (UniqueName: \"kubernetes.io/projected/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-kube-api-access-f57vq\") pod \"maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:36.908836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.908816 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:36.916742 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:36.916722 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:37.307702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:37.307679 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m"] Oct 30 03:19:37.308903 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:19:37.308874 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c64fa97_bddf_40ae_87a1_2b4d97dd9cb5.slice/crio-a41f70374aa173fa44b1b259a5c623e9a696ed38d85f204b67e3b22d09b6e1d9 WatchSource:0}: Error finding container a41f70374aa173fa44b1b259a5c623e9a696ed38d85f204b67e3b22d09b6e1d9: Status 404 returned error can't find the container with id a41f70374aa173fa44b1b259a5c623e9a696ed38d85f204b67e3b22d09b6e1d9 Oct 30 03:19:37.907284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:37.907253 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" event={"ID":"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5","Type":"ContainerStarted","Data":"fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475"} Oct 30 03:19:37.907665 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:37.907309 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:37.907665 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:37.907327 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" event={"ID":"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5","Type":"ContainerStarted","Data":"a41f70374aa173fa44b1b259a5c623e9a696ed38d85f204b67e3b22d09b6e1d9"} Oct 30 03:19:37.954099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:37.953997 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" podStartSLOduration=1.561913803 podStartE2EDuration="1.953984254s" totalImagesPullingTime="392.070451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:37.952100643 +0000 UTC m=+1001.954453837" watchObservedRunningTime="2026-10-30 03:19:37.953984254 +0000 UTC m=+1001.956337450" Oct 30 03:19:38.546979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:38.546949 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:40.228238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:40.228198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:41.974961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:41.974927 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:43.673475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:43.673440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:43.917219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:43.917188 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:19:43.942767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:43.942683 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:45.277411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:45.277376 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:45.645527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:45.645443 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:46.414173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:46.414140 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:46.820530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:46.820482 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:46.926311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:46.926280 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:47.034020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:47.033978 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:48.733475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:48.733429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:50.818109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:50.818054 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:50.975946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:50.975910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:51.223222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:51.223132 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:52.647395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:52.647351 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:52.746657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:52.746626 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:52.835825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:52.835787 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:54.669012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:54.668970 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:54.860874 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:19:54.860800 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:19:56.417460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:56.417423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:57.630889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:57.630849 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:58.050238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.050201 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:58.073717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.073684 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerID="3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e" exitCode=0 Oct 30 03:19:58.073901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.073731 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" event={"ID":"9e3313fb-3e34-42bc-a4b5-aa52df7756ec","Type":"ContainerDied","Data":"3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e"} Oct 30 03:19:58.074139 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.074124 4442 scope.go:122] "RemoveContainer" containerID="3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e" Oct 30 03:19:58.075436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.075407 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:58.075529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.075475 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:58.075529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.075504 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:19:58.305993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.305903 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:19:58.316350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.316326 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:58.810275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:58.810203 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:59.010506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:59.010466 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:19:59.091111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:59.091017 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" event={"ID":"9e3313fb-3e34-42bc-a4b5-aa52df7756ec","Type":"ContainerStarted","Data":"53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00"} Oct 30 03:19:59.679163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:59.679129 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:19:59.679611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:59.679446 4442 patch_prober.go:29] interesting pod/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 03:19:59.683834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:59.680843 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 03:19:59.974312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:19:59.974228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:00.673224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:00.673177 4442 patch_prober.go:29] interesting pod/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 03:20:00.673429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:00.673247 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 03:20:01.672745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:01.672710 4442 patch_prober.go:29] interesting pod/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 03:20:01.673160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:01.672790 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 03:20:01.935827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:01.935716 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:02.037995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:02.037961 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:02.152341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:02.152309 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:02.247308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:02.247216 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:02.672791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:02.672726 4442 patch_prober.go:29] interesting pod/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 03:20:02.673332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:02.672835 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 03:20:03.672312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.672273 4442 patch_prober.go:29] interesting pod/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 03:20:03.672499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.672340 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 03:20:03.710214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.710075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:03.811859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.810298 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:03.815593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.815561 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr"] Oct 30 03:20:03.820184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.820164 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:03.839972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.839947 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr"] Oct 30 03:20:03.847979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.847956 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cc0859\"" Oct 30 03:20:03.859733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.859707 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cxz\" (UniqueName: \"kubernetes.io/projected/89212ab1-68bb-4e18-a406-a997324528ec-kube-api-access-l7cxz\") pod \"payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:03.859866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.859767 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89212ab1-68bb-4e18-a406-a997324528ec-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:03.963918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.963830 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89212ab1-68bb-4e18-a406-a997324528ec-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:03.971618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.971560 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cxz\" (UniqueName: \"kubernetes.io/projected/89212ab1-68bb-4e18-a406-a997324528ec-kube-api-access-l7cxz\") pod \"payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:03.972563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:03.970784 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89212ab1-68bb-4e18-a406-a997324528ec-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:04.018815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:04.018782 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cxz\" (UniqueName: \"kubernetes.io/projected/89212ab1-68bb-4e18-a406-a997324528ec-kube-api-access-l7cxz\") pod \"payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:04.076398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:04.076340 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:04.133817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:04.133788 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:04.487314 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:04.487288 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr"] Oct 30 03:20:04.488987 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:20:04.488929 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89212ab1_68bb_4e18_a406_a997324528ec.slice/crio-dc11ec9d201c222d85fb0e8c23a8a0e50431ce4b50cc05fc05e34ecf282db32c WatchSource:0}: Error finding container dc11ec9d201c222d85fb0e8c23a8a0e50431ce4b50cc05fc05e34ecf282db32c: Status 404 returned error can't find the container with id dc11ec9d201c222d85fb0e8c23a8a0e50431ce4b50cc05fc05e34ecf282db32c Oct 30 03:20:04.676704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:04.676678 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:20:04.677154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:04.677126 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:20:04.677949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:04.677930 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:20:05.151497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:05.151455 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" event={"ID":"89212ab1-68bb-4e18-a406-a997324528ec","Type":"ContainerStarted","Data":"f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70"} Oct 30 03:20:05.151933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:05.151539 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" event={"ID":"89212ab1-68bb-4e18-a406-a997324528ec","Type":"ContainerStarted","Data":"dc11ec9d201c222d85fb0e8c23a8a0e50431ce4b50cc05fc05e34ecf282db32c"} Oct 30 03:20:05.151933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:05.151561 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:05.188939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:05.188886 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" podStartSLOduration=2.188873681 podStartE2EDuration="2.188873681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:05.186034709 +0000 UTC m=+1029.188387917" watchObservedRunningTime="2026-10-30 03:20:05.188873681 +0000 UTC m=+1029.191226878" Oct 30 03:20:05.820294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:05.820260 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:07.533465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.533426 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt"] Oct 30 03:20:07.537297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.537282 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.566034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.566004 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3e4daf-serving-cert\"" Oct 30 03:20:07.568669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.568645 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt"] Oct 30 03:20:07.628568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.628542 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-maas-api-tls\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.628724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.628707 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a09fae0-0791-42f7-900a-a7439869821a-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.628801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.628735 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-metrics-certs\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.628836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.628809 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngqdr\" (UniqueName: \"kubernetes.io/projected/6a09fae0-0791-42f7-900a-a7439869821a-kube-api-access-ngqdr\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.730216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.730185 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a09fae0-0791-42f7-900a-a7439869821a-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.730216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.730218 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-metrics-certs\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.730423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.730238 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngqdr\" (UniqueName: \"kubernetes.io/projected/6a09fae0-0791-42f7-900a-a7439869821a-kube-api-access-ngqdr\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.730423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.730288 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-maas-api-tls\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.730884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.730850 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a09fae0-0791-42f7-900a-a7439869821a-openshift-service-ca\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.732817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.732796 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-maas-api-tls\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.732919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.732841 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-metrics-certs\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.762709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.762679 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:07.763648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.763623 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngqdr\" (UniqueName: \"kubernetes.io/projected/6a09fae0-0791-42f7-900a-a7439869821a-kube-api-access-ngqdr\") pod \"maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.850492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.850409 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:07.886330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:07.886298 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:08.173120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:08.173042 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt"] Oct 30 03:20:08.180549 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:20:08.180508 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a09fae0_0791_42f7_900a_a7439869821a.slice/crio-d62308febda5bbfb686841183d7204456dd76ab366334dabd441520d3570b5fb WatchSource:0}: Error finding container d62308febda5bbfb686841183d7204456dd76ab366334dabd441520d3570b5fb: Status 404 returned error can't find the container with id d62308febda5bbfb686841183d7204456dd76ab366334dabd441520d3570b5fb Oct 30 03:20:08.249391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:08.249364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:08.391988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:08.391742 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:09.176208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:09.176124 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" event={"ID":"6a09fae0-0791-42f7-900a-a7439869821a","Type":"ContainerStarted","Data":"e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95"} Oct 30 03:20:09.176208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:09.176181 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" event={"ID":"6a09fae0-0791-42f7-900a-a7439869821a","Type":"ContainerStarted","Data":"d62308febda5bbfb686841183d7204456dd76ab366334dabd441520d3570b5fb"} Oct 30 03:20:09.176710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:09.176244 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:09.220813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:09.220748 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" podStartSLOduration=1.7929976490000001 podStartE2EDuration="2.220731252s" totalImagesPullingTime="427.733603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:09.218792377 +0000 UTC m=+1033.221145599" watchObservedRunningTime="2026-10-30 03:20:09.220731252 +0000 UTC m=+1033.223084451" Oct 30 03:20:09.744307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:09.744232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:11.541504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:11.541454 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:12.151444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:12.151415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:12.272552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:12.272513 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:13.635099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:13.635068 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:14.018222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:14.018195 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:14.130915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:14.130871 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:15.186145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:15.186115 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:20:15.539980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:15.539940 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:16.159436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:16.159403 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:20:17.380438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:17.380408 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:18.371784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:18.370561 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:20:19.225327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:19.225287 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:20.809393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.809361 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r"] Oct 30 03:20:20.813406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.813389 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.838040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.838014 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.838181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.838049 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-metrics-certs\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.838181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.838083 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.838406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.838378 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56zg\" (UniqueName: \"kubernetes.io/projected/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-kube-api-access-r56zg\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.838669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.838470 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cc0859-serving-cert\"" Oct 30 03:20:20.843098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.843075 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r"] Oct 30 03:20:20.887092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.887065 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:20.940194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.940169 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56zg\" (UniqueName: \"kubernetes.io/projected/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-kube-api-access-r56zg\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.940343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.940239 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.940343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.940262 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-metrics-certs\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.940343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.940283 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.940504 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:20.940389 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-serving-cert: secret "maas-api-e2e-api-b-cc0859-serving-cert" not found Oct 30 03:20:20.940504 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:20.940454 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls podName:8acd754c-5d8b-4a1a-9efb-68448dbb7fa9 nodeName:}" failed. No retries permitted until 2026-10-30 03:20:21.440439318 +0000 UTC m=+1045.442792493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls") pod "maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" (UID: "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9") : secret "maas-api-e2e-api-b-cc0859-serving-cert" not found Oct 30 03:20:20.940861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.940843 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-openshift-service-ca\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.942870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.942848 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-metrics-certs\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.960451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.960421 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56zg\" (UniqueName: \"kubernetes.io/projected/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-kube-api-access-r56zg\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:20.985280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:20.985254 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:21.128238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:21.128116 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:21.448507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:21.448430 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:21.450977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:21.450947 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls\") pod \"maas-api-e2e-api-b-cc0859-67976b5759-bbt4r\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:21.727325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:21.727242 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:22.039282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:22.039253 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r"] Oct 30 03:20:22.040929 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:20:22.040895 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acd754c_5d8b_4a1a_9efb_68448dbb7fa9.slice/crio-70fb7764b8097d59ac47453e9bf34b65334cf4dd1dbc3d004f24ea4679350629 WatchSource:0}: Error finding container 70fb7764b8097d59ac47453e9bf34b65334cf4dd1dbc3d004f24ea4679350629: Status 404 returned error can't find the container with id 70fb7764b8097d59ac47453e9bf34b65334cf4dd1dbc3d004f24ea4679350629 Oct 30 03:20:22.281221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:22.281182 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" event={"ID":"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9","Type":"ContainerStarted","Data":"70fb7764b8097d59ac47453e9bf34b65334cf4dd1dbc3d004f24ea4679350629"} Oct 30 03:20:22.718976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:22.718897 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:23.288318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:23.288282 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" event={"ID":"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9","Type":"ContainerStarted","Data":"ffd485b9d213fd924b549b1712fd0ab759636ff526560de5391562c84c321aed"} Oct 30 03:20:23.288832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:23.288468 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:23.329549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:23.329496 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" podStartSLOduration=3.057022393 podStartE2EDuration="3.329482006s" totalImagesPullingTime="272.459613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:23.326622911 +0000 UTC m=+1047.328976108" watchObservedRunningTime="2026-10-30 03:20:23.329482006 +0000 UTC m=+1047.331835204" Oct 30 03:20:23.643998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:23.643914 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg"] Oct 30 03:20:23.741272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:23.741238 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:23.865896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:23.865847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:24.496713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:24.496684 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:24.860076 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:24.860044 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:20:26.297627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:26.297588 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:27.913359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:27.913327 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:28.644770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:28.644693 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" podUID="c7cd6588-486b-411a-8365-fc57c97da59f" containerName="payload-pre-processing" containerID="cri-o://fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36" gracePeriod=25 Oct 30 03:20:29.302648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.302621 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:20:29.564773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.564674 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" podUID="c7cd6588-486b-411a-8365-fc57c97da59f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:20:29.620193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.620157 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb"] Oct 30 03:20:29.624346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.624323 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.643084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.643064 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3b93b5-serving-cert\"" Oct 30 03:20:29.643734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.643705 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb"] Oct 30 03:20:29.676094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.676054 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.676348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.676320 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-openshift-service-ca\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.676454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.676366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-metrics-certs\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.676454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.676393 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwmk\" (UniqueName: \"kubernetes.io/projected/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-kube-api-access-vhwmk\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.738329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.738298 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:29.777823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.777787 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-openshift-service-ca\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.777823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.777827 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-metrics-certs\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.778091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.777854 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwmk\" (UniqueName: \"kubernetes.io/projected/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-kube-api-access-vhwmk\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.778091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.777938 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.778091 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:29.778081 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-serving-cert: secret "maas-api-e2e-api-a-3b93b5-serving-cert" not found Oct 30 03:20:29.778270 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:29.778162 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls podName:d6d4e3e2-f826-4f72-8698-a56ff7211ae5 nodeName:}" failed. No retries permitted until 2026-10-30 03:20:30.278140495 +0000 UTC m=+1054.280493684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls") pod "maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" (UID: "d6d4e3e2-f826-4f72-8698-a56ff7211ae5") : secret "maas-api-e2e-api-a-3b93b5-serving-cert" not found Oct 30 03:20:29.778444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.778427 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-openshift-service-ca\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.780566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.780533 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-metrics-certs\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.798055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.798029 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwmk\" (UniqueName: \"kubernetes.io/projected/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-kube-api-access-vhwmk\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:29.879936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:29.879740 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:30.042687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:30.042653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:30.283506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:30.283469 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:30.286117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:30.286094 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls\") pod \"maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:30.537023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:30.536938 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:30.925611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:30.923160 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb"] Oct 30 03:20:31.347306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:31.347269 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" event={"ID":"d6d4e3e2-f826-4f72-8698-a56ff7211ae5","Type":"ContainerStarted","Data":"a60394243a378a5c1aad0586aec9d5f144b5103545cbbefe9c2d11392f218577"} Oct 30 03:20:31.609305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:31.609217 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:31.713929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:31.713895 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:32.354451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:32.354413 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" event={"ID":"d6d4e3e2-f826-4f72-8698-a56ff7211ae5","Type":"ContainerStarted","Data":"6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201"} Oct 30 03:20:32.354636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:32.354560 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:32.394237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:32.394184 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" podStartSLOduration=3.009666134 podStartE2EDuration="3.394171756s" totalImagesPullingTime="384.505622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:20:32.392122856 +0000 UTC m=+1056.394476092" watchObservedRunningTime="2026-10-30 03:20:32.394171756 +0000 UTC m=+1056.396524952" Oct 30 03:20:33.353931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:33.353898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:34.357980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:34.357950 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:34.617188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:34.617106 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:35.008300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:35.008269 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:36.840599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:36.840559 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:38.365028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:38.364995 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:20:38.547903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:38.547877 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:39.564797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:39.564732 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" podUID="c7cd6588-486b-411a-8365-fc57c97da59f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:20:39.624418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:39.624387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:40.186324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:40.186289 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf"] Oct 30 03:20:40.243596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:40.243558 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:40.346330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:40.346294 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:40.733039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:40.731370 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:42.074899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:42.074857 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:42.546859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:42.546830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:44.023830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:44.023794 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:44.126084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:44.126055 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:44.357983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:44.357904 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:45.186830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.186788 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" podUID="e5cd4573-0ed7-40a3-bb2a-c01060e9021a" containerName="payload-pre-processing" containerID="cri-o://15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11" gracePeriod=25 Oct 30 03:20:45.447556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.447502 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:20:45.457185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.457158 4442 generic.go:356] "Generic (PLEG): container finished" podID="e5cd4573-0ed7-40a3-bb2a-c01060e9021a" containerID="15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11" exitCode=0 Oct 30 03:20:45.457304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.457202 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" Oct 30 03:20:45.457304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.457244 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" event={"ID":"e5cd4573-0ed7-40a3-bb2a-c01060e9021a","Type":"ContainerDied","Data":"15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11"} Oct 30 03:20:45.457435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.457304 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf" event={"ID":"e5cd4573-0ed7-40a3-bb2a-c01060e9021a","Type":"ContainerDied","Data":"753b3e9c705a9b4133d950f94139eb6e904407f961cbcdd7c320c7e2f820640d"} Oct 30 03:20:45.457435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.457334 4442 scope.go:122] "RemoveContainer" containerID="15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11" Oct 30 03:20:45.466808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.466785 4442 scope.go:122] "RemoveContainer" containerID="15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11" Oct 30 03:20:45.467080 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:45.467055 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11\": container with ID starting with 15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11 not found: ID does not exist" containerID="15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11" Oct 30 03:20:45.467138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.467079 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11"} err="failed to get container status \"15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11\": rpc error: code = NotFound desc = could not find container \"15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11\": container with ID starting with 15587f313994a5e3a6533fff56d81d33a525b4983aed0d3761ae976d41f56d11 not found: ID does not exist" Oct 30 03:20:45.533383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.533344 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-plugins-config-volume\") pod \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " Oct 30 03:20:45.533383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.533382 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-kube-api-access-wst65\" (UniqueName: \"kubernetes.io/projected/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-kube-api-access-wst65\") pod \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\" (UID: \"e5cd4573-0ed7-40a3-bb2a-c01060e9021a\") " Oct 30 03:20:45.533712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.533688 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-plugins-config-volume" pod "e5cd4573-0ed7-40a3-bb2a-c01060e9021a" (UID: "e5cd4573-0ed7-40a3-bb2a-c01060e9021a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:20:45.533821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.533809 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:45.535687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.535665 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-kube-api-access-wst65" pod "e5cd4573-0ed7-40a3-bb2a-c01060e9021a" (UID: "e5cd4573-0ed7-40a3-bb2a-c01060e9021a"). InnerVolumeSpecName "kube-api-access-wst65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:45.634511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.634481 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wst65\" (UniqueName: \"kubernetes.io/projected/e5cd4573-0ed7-40a3-bb2a-c01060e9021a-kube-api-access-wst65\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:45.804257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.804203 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf"] Oct 30 03:20:45.815277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:45.815251 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f247d1-5f878678f5-4kscf"] Oct 30 03:20:46.078510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:46.078429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:46.111601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:46.111564 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5cd4573-0ed7-40a3-bb2a-c01060e9021a" path="/var/lib/kubelet/pods/e5cd4573-0ed7-40a3-bb2a-c01060e9021a/volumes" Oct 30 03:20:47.232177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:47.232147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:47.707905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:47.707873 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:47.947860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:47.947824 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:48.332999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:48.332934 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:20:48.966078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:48.965965 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:20:48.976483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:48.975277 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7cd6588-486b-411a-8365-fc57c97da59f-kube-api-access-nffwd\" (UniqueName: \"kubernetes.io/projected/c7cd6588-486b-411a-8365-fc57c97da59f-kube-api-access-nffwd\") pod \"c7cd6588-486b-411a-8365-fc57c97da59f\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " Oct 30 03:20:48.976483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:48.975393 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7cd6588-486b-411a-8365-fc57c97da59f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cd6588-486b-411a-8365-fc57c97da59f-plugins-config-volume\") pod \"c7cd6588-486b-411a-8365-fc57c97da59f\" (UID: \"c7cd6588-486b-411a-8365-fc57c97da59f\") " Oct 30 03:20:48.991110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:48.983103 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7cd6588-486b-411a-8365-fc57c97da59f-plugins-config-volume" pod "c7cd6588-486b-411a-8365-fc57c97da59f" (UID: "c7cd6588-486b-411a-8365-fc57c97da59f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:20:48.991110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:48.989801 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7cd6588-486b-411a-8365-fc57c97da59f-kube-api-access-nffwd" pod "c7cd6588-486b-411a-8365-fc57c97da59f" (UID: "c7cd6588-486b-411a-8365-fc57c97da59f"). InnerVolumeSpecName "kube-api-access-nffwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:49.076703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.076668 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffwd\" (UniqueName: \"kubernetes.io/projected/c7cd6588-486b-411a-8365-fc57c97da59f-kube-api-access-nffwd\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:49.076703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.076697 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7cd6588-486b-411a-8365-fc57c97da59f-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:49.473155 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.473105 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss"] Oct 30 03:20:49.500280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.500239 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" podUID="2da698ab-d12f-4add-8361-3a2ddf970672" containerName="maas-api" containerID="cri-o://90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2" gracePeriod=30 Oct 30 03:20:49.595791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.589959 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:49.595791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.593961 4442 generic.go:356] "Generic (PLEG): container finished" podID="c7cd6588-486b-411a-8365-fc57c97da59f" containerID="fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36" exitCode=0 Oct 30 03:20:49.595791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.594056 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" Oct 30 03:20:49.595791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.594115 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" event={"ID":"c7cd6588-486b-411a-8365-fc57c97da59f","Type":"ContainerDied","Data":"fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36"} Oct 30 03:20:49.595791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.594160 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg" event={"ID":"c7cd6588-486b-411a-8365-fc57c97da59f","Type":"ContainerDied","Data":"52442c5d4c6125092923c42a807f7ebf4e7c7b422f518bcf206dc1c936c7ffe9"} Oct 30 03:20:49.595791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.594189 4442 scope.go:122] "RemoveContainer" containerID="fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36" Oct 30 03:20:49.638954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.638928 4442 scope.go:122] "RemoveContainer" containerID="fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36" Oct 30 03:20:49.639375 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:49.639350 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36\": container with ID starting with fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36 not found: ID does not exist" containerID="fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36" Oct 30 03:20:49.639461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.639387 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36"} err="failed to get container status \"fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36\": rpc error: code = NotFound desc = could not find container \"fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36\": container with ID starting with fdeb7c84af3e709431a6109ed9e8bef5526397d29e0867663ffdb5bef18b8b36 not found: ID does not exist" Oct 30 03:20:49.654924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.654890 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg"] Oct 30 03:20:49.658972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:49.658942 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b132360f-55d7d7d6b7-vxxxg"] Oct 30 03:20:50.025872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.025839 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:20:50.046605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.046574 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-maas-api-tls\") pod \"2da698ab-d12f-4add-8361-3a2ddf970672\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " Oct 30 03:20:50.046738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.046653 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da698ab-d12f-4add-8361-3a2ddf970672-kube-api-access-76xq2\" (UniqueName: \"kubernetes.io/projected/2da698ab-d12f-4add-8361-3a2ddf970672-kube-api-access-76xq2\") pod \"2da698ab-d12f-4add-8361-3a2ddf970672\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " Oct 30 03:20:50.046738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.046702 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-metrics-certs\") pod \"2da698ab-d12f-4add-8361-3a2ddf970672\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " Oct 30 03:20:50.046901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.046797 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da698ab-d12f-4add-8361-3a2ddf970672-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da698ab-d12f-4add-8361-3a2ddf970672-openshift-service-ca\") pod \"2da698ab-d12f-4add-8361-3a2ddf970672\" (UID: \"2da698ab-d12f-4add-8361-3a2ddf970672\") " Oct 30 03:20:50.047242 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.047213 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da698ab-d12f-4add-8361-3a2ddf970672-openshift-service-ca" pod "2da698ab-d12f-4add-8361-3a2ddf970672" (UID: "2da698ab-d12f-4add-8361-3a2ddf970672"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:20:50.049167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.049130 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da698ab-d12f-4add-8361-3a2ddf970672-kube-api-access-76xq2" pod "2da698ab-d12f-4add-8361-3a2ddf970672" (UID: "2da698ab-d12f-4add-8361-3a2ddf970672"). InnerVolumeSpecName "kube-api-access-76xq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:50.049322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.049296 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-maas-api-tls" pod "2da698ab-d12f-4add-8361-3a2ddf970672" (UID: "2da698ab-d12f-4add-8361-3a2ddf970672"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:50.049533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.049507 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-metrics-certs" pod "2da698ab-d12f-4add-8361-3a2ddf970672" (UID: "2da698ab-d12f-4add-8361-3a2ddf970672"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:50.109439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.109398 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7cd6588-486b-411a-8365-fc57c97da59f" path="/var/lib/kubelet/pods/c7cd6588-486b-411a-8365-fc57c97da59f/volumes" Oct 30 03:20:50.128053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.128023 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:50.147926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.147900 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2da698ab-d12f-4add-8361-3a2ddf970672-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:50.147926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.147922 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:50.148156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.147932 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xq2\" (UniqueName: \"kubernetes.io/projected/2da698ab-d12f-4add-8361-3a2ddf970672-kube-api-access-76xq2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:50.148156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.147941 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2da698ab-d12f-4add-8361-3a2ddf970672-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:50.237424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.237388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:50.361189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.361101 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:50.601632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.601599 4442 generic.go:356] "Generic (PLEG): container finished" podID="2da698ab-d12f-4add-8361-3a2ddf970672" containerID="90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2" exitCode=0 Oct 30 03:20:50.602132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.601653 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" Oct 30 03:20:50.602132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.601684 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" event={"ID":"2da698ab-d12f-4add-8361-3a2ddf970672","Type":"ContainerDied","Data":"90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2"} Oct 30 03:20:50.602132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.601767 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss" event={"ID":"2da698ab-d12f-4add-8361-3a2ddf970672","Type":"ContainerDied","Data":"b61628494c9c5b3e3f1dfccdbab75f19a90c847a6098a07937f76514ab483e56"} Oct 30 03:20:50.602132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.601799 4442 scope.go:122] "RemoveContainer" containerID="90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2" Oct 30 03:20:50.614715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.614694 4442 scope.go:122] "RemoveContainer" containerID="90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2" Oct 30 03:20:50.615150 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:50.615117 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2\": container with ID starting with 90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2 not found: ID does not exist" containerID="90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2" Oct 30 03:20:50.615234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.615164 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2"} err="failed to get container status \"90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2\": rpc error: code = NotFound desc = could not find container \"90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2\": container with ID starting with 90b1cafa335c046d81acacbfb1d4d441029db9bb58b6811dbe416a1b10b6e0d2 not found: ID does not exist" Oct 30 03:20:50.650826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.650783 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss"] Oct 30 03:20:50.661832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:50.661803 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-kk6ss"] Oct 30 03:20:51.061793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:51.061728 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:51.211637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:51.211583 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:51.655796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:51.651924 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:52.115047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.114531 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da698ab-d12f-4add-8361-3a2ddf970672" path="/var/lib/kubelet/pods/2da698ab-d12f-4add-8361-3a2ddf970672/volumes" Oct 30 03:20:52.172355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.172311 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m"] Oct 30 03:20:52.172585 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.172560 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" podUID="0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" containerName="maas-api" containerID="cri-o://fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475" gracePeriod=30 Oct 30 03:20:52.279806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.279658 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:52.447743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.447715 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:20:52.634485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.634441 4442 generic.go:356] "Generic (PLEG): container finished" podID="0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" containerID="fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475" exitCode=0 Oct 30 03:20:52.634837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.634659 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" event={"ID":"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5","Type":"ContainerDied","Data":"fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475"} Oct 30 03:20:52.634837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.634713 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" event={"ID":"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5","Type":"ContainerDied","Data":"a41f70374aa173fa44b1b259a5c623e9a696ed38d85f204b67e3b22d09b6e1d9"} Oct 30 03:20:52.634837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.634746 4442 scope.go:122] "RemoveContainer" containerID="fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475" Oct 30 03:20:52.635607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.635189 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-maas-api-tls\") pod \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " Oct 30 03:20:52.635607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.635268 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-metrics-certs\") pod \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " Oct 30 03:20:52.635607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.635324 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-kube-api-access-f57vq\" (UniqueName: \"kubernetes.io/projected/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-kube-api-access-f57vq\") pod \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " Oct 30 03:20:52.635607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.635424 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-openshift-service-ca\") pod \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\" (UID: \"0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5\") " Oct 30 03:20:52.662491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.636145 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-openshift-service-ca" pod "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" (UID: "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:20:52.662491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.636233 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m" Oct 30 03:20:52.662491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.651198 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-maas-api-tls" pod "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" (UID: "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:52.674481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.673933 4442 scope.go:122] "RemoveContainer" containerID="fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475" Oct 30 03:20:52.674481 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:52.674396 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475\": container with ID starting with fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475 not found: ID does not exist" containerID="fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475" Oct 30 03:20:52.674481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.674435 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475"} err="failed to get container status \"fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475\": rpc error: code = NotFound desc = could not find container \"fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475\": container with ID starting with fabf7d1f310549bd80317ad83ccc6410e932c3182d2870e1ae8e4eb9a4a9b475 not found: ID does not exist" Oct 30 03:20:52.682427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.682376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-metrics-certs" pod "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" (UID: "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:52.740166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.740124 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:52.740166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.740166 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:52.740410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.740188 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:52.754926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.754882 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-kube-api-access-f57vq" pod "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" (UID: "0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5"). InnerVolumeSpecName "kube-api-access-f57vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:52.764795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.762592 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm"] Oct 30 03:20:52.764795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.762878 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" podUID="359cbe0c-3723-4266-8fa5-fc5b2a0a63da" containerName="maas-api" containerID="cri-o://b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8" gracePeriod=30 Oct 30 03:20:52.781869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.781739 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:52.840844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.840813 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f57vq\" (UniqueName: \"kubernetes.io/projected/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5-kube-api-access-f57vq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:52.984688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.984647 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m"] Oct 30 03:20:52.984847 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:52.984714 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-585938-56bdbd6f5b-n6l5m"] Oct 30 03:20:53.042197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.042172 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:20:53.142915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.142879 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-maas-api-tls\") pod \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " Oct 30 03:20:53.143100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.142963 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-openshift-service-ca\") pod \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " Oct 30 03:20:53.143100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.143071 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-metrics-certs\") pod \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " Oct 30 03:20:53.143218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.143161 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-kube-api-access-bhnzv\" (UniqueName: \"kubernetes.io/projected/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-kube-api-access-bhnzv\") pod \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\" (UID: \"359cbe0c-3723-4266-8fa5-fc5b2a0a63da\") " Oct 30 03:20:53.143466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.143435 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-openshift-service-ca" pod "359cbe0c-3723-4266-8fa5-fc5b2a0a63da" (UID: "359cbe0c-3723-4266-8fa5-fc5b2a0a63da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:20:53.145649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.145619 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-metrics-certs" pod "359cbe0c-3723-4266-8fa5-fc5b2a0a63da" (UID: "359cbe0c-3723-4266-8fa5-fc5b2a0a63da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:53.146089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.146067 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-kube-api-access-bhnzv" pod "359cbe0c-3723-4266-8fa5-fc5b2a0a63da" (UID: "359cbe0c-3723-4266-8fa5-fc5b2a0a63da"). InnerVolumeSpecName "kube-api-access-bhnzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:20:53.146413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.146397 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-maas-api-tls" pod "359cbe0c-3723-4266-8fa5-fc5b2a0a63da" (UID: "359cbe0c-3723-4266-8fa5-fc5b2a0a63da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:20:53.245142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.244822 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhnzv\" (UniqueName: \"kubernetes.io/projected/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-kube-api-access-bhnzv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:53.245142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.244858 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:53.245142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.244873 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:53.245142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.244887 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/359cbe0c-3723-4266-8fa5-fc5b2a0a63da-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:20:53.619116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.619083 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:53.641060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.641027 4442 generic.go:356] "Generic (PLEG): container finished" podID="359cbe0c-3723-4266-8fa5-fc5b2a0a63da" containerID="b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8" exitCode=0 Oct 30 03:20:53.641221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.641077 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" Oct 30 03:20:53.641221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.641117 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" event={"ID":"359cbe0c-3723-4266-8fa5-fc5b2a0a63da","Type":"ContainerDied","Data":"b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8"} Oct 30 03:20:53.641221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.641169 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm" event={"ID":"359cbe0c-3723-4266-8fa5-fc5b2a0a63da","Type":"ContainerDied","Data":"42696cdee67ab229fa7e7176620c1265cce27701e19d072e92acfa96c322fcd3"} Oct 30 03:20:53.641221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.641195 4442 scope.go:122] "RemoveContainer" containerID="b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8" Oct 30 03:20:53.654725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.654691 4442 scope.go:122] "RemoveContainer" containerID="b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8" Oct 30 03:20:53.655030 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:53.655010 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8\": container with ID starting with b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8 not found: ID does not exist" containerID="b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8" Oct 30 03:20:53.655107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.655040 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8"} err="failed to get container status \"b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8\": rpc error: code = NotFound desc = could not find container \"b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8\": container with ID starting with b2efa70c94946d146721cee849b9c2dec2cf9d01e8c34f459b469fd3890191e8 not found: ID does not exist" Oct 30 03:20:53.711848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.711808 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm"] Oct 30 03:20:53.720589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:53.720554 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f247d1-794db7449d-dbbjm"] Oct 30 03:20:54.106149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:54.106119 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" path="/var/lib/kubelet/pods/0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5/volumes" Oct 30 03:20:54.106507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:54.106494 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359cbe0c-3723-4266-8fa5-fc5b2a0a63da" path="/var/lib/kubelet/pods/359cbe0c-3723-4266-8fa5-fc5b2a0a63da/volumes" Oct 30 03:20:54.296604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:54.296569 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:54.539303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:54.539276 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:54.744225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:54.744186 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:54.860866 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:20:54.860744 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:20:55.022381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:55.022349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:55.423223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:55.423182 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:55.539348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:55.539272 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:56.227807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:56.227745 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:56.517131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:56.517089 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql"] Oct 30 03:20:56.517420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:56.517392 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" containerID="cri-o://ee5a0b546004cae491639ecf24fe0b9ef4000a06802b4f6dd0ff7d9e1202aaa1" gracePeriod=30 Oct 30 03:20:57.253361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:57.253294 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:57.686932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:57.686847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:57.828473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:57.827218 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:57.942170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:57.942071 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:58.087075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:58.087042 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:58.531603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:58.531561 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:58.774192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:58.774162 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:58.881442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:58.881364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:59.195805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.195702 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:59.258841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.258805 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj"] Oct 30 03:20:59.262852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.262818 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" containerName="maas-api" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.262904 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" containerName="maas-api" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.262934 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2da698ab-d12f-4add-8361-3a2ddf970672" containerName="maas-api" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.262942 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da698ab-d12f-4add-8361-3a2ddf970672" containerName="maas-api" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.262954 4442 cpu_manager.go:414] "containerMap: removing container" podUID="359cbe0c-3723-4266-8fa5-fc5b2a0a63da" containerName="maas-api" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.262961 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359cbe0c-3723-4266-8fa5-fc5b2a0a63da" containerName="maas-api" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263018 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c7cd6588-486b-411a-8365-fc57c97da59f" containerName="payload-pre-processing" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263026 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7cd6588-486b-411a-8365-fc57c97da59f" containerName="payload-pre-processing" Oct 30 03:20:59.263040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263042 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e5cd4573-0ed7-40a3-bb2a-c01060e9021a" containerName="payload-pre-processing" Oct 30 03:20:59.263471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263049 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5cd4573-0ed7-40a3-bb2a-c01060e9021a" containerName="payload-pre-processing" Oct 30 03:20:59.263471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263174 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7cd6588-486b-411a-8365-fc57c97da59f" containerName="payload-pre-processing" Oct 30 03:20:59.263471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263185 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5cd4573-0ed7-40a3-bb2a-c01060e9021a" containerName="payload-pre-processing" Oct 30 03:20:59.263471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263194 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c64fa97-bddf-40ae-87a1-2b4d97dd9cb5" containerName="maas-api" Oct 30 03:20:59.263471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263204 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da698ab-d12f-4add-8361-3a2ddf970672" containerName="maas-api" Oct 30 03:20:59.263471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.263212 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="359cbe0c-3723-4266-8fa5-fc5b2a0a63da" containerName="maas-api" Oct 30 03:20:59.285534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.280964 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj"] Oct 30 03:20:59.285534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.281012 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq"] Oct 30 03:20:59.285534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.281220 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" containerID="cri-o://53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00" gracePeriod=30 Oct 30 03:20:59.285534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.281424 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.305213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.304911 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d4b0d5-openshift-default-dockercfg-cs2q7\"" Oct 30 03:20:59.319211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.319181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.465867 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83237762-f446-40ca-9500-861aaf8a326e-istio-podinfo\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.465981 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-socket\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.466021 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-certs\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.466051 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-data\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.466075 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-envoy\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.466110 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dflp\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-kube-api-access-2dflp\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.466152 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-credential-socket\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.466193 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83237762-f446-40ca-9500-861aaf8a326e-istiod-ca-cert\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.485276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.466223 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-istio-token\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.508634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.492961 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:20:59.567376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567337 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-data\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567373 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-envoy\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dflp\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-kube-api-access-2dflp\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567436 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-credential-socket\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567475 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83237762-f446-40ca-9500-861aaf8a326e-istiod-ca-cert\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-istio-token\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567590 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83237762-f446-40ca-9500-861aaf8a326e-istio-podinfo\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567697 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-socket\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567734 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-certs\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.567936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.567809 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-data\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.568317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.568066 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-socket\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.568317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.568095 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-credential-socket\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.568317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.568154 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-certs\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.568317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.568231 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83237762-f446-40ca-9500-861aaf8a326e-istiod-ca-cert\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.569931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.569908 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-envoy\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.570423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.570403 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83237762-f446-40ca-9500-861aaf8a326e-istio-podinfo\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.585810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.585788 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-istio-token\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.585928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.585871 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dflp\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-kube-api-access-2dflp\") pod \"e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.617510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.617483 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:20:59.930960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.930933 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj"] Oct 30 03:20:59.931305 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:20:59.931282 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83237762_f446_40ca_9500_861aaf8a326e.slice/crio-701f3c1c3053c0eb5c5349e9c3df38594b5a9bcf54393268ae5c0ac20f313549 WatchSource:0}: Error finding container 701f3c1c3053c0eb5c5349e9c3df38594b5a9bcf54393268ae5c0ac20f313549: Status 404 returned error can't find the container with id 701f3c1c3053c0eb5c5349e9c3df38594b5a9bcf54393268ae5c0ac20f313549 Oct 30 03:20:59.933796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.933766 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:20:59.933867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.933832 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:20:59.933902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:20:59.933891 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:21:00.196796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:00.196662 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:00.689020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:00.688987 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" event={"ID":"83237762-f446-40ca-9500-861aaf8a326e","Type":"ContainerStarted","Data":"49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b"} Oct 30 03:21:00.689418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:00.689045 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" event={"ID":"83237762-f446-40ca-9500-861aaf8a326e","Type":"ContainerStarted","Data":"701f3c1c3053c0eb5c5349e9c3df38594b5a9bcf54393268ae5c0ac20f313549"} Oct 30 03:21:00.751808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:00.751738 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" podStartSLOduration=1.751723449 podStartE2EDuration="1.751723449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:00.739480203 +0000 UTC m=+1084.741833397" watchObservedRunningTime="2026-10-30 03:21:00.751723449 +0000 UTC m=+1084.754076645" Oct 30 03:21:01.318926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.318891 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:01.618572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.618488 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:01.626236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.626206 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:01.636082 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.636052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:01.700962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.700934 4442 generic.go:356] "Generic (PLEG): container finished" podID="9d818414-c583-416a-9561-927249e337e5" containerID="ee5a0b546004cae491639ecf24fe0b9ef4000a06802b4f6dd0ff7d9e1202aaa1" exitCode=0 Oct 30 03:21:01.701410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.701002 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" event={"ID":"9d818414-c583-416a-9561-927249e337e5","Type":"ContainerDied","Data":"ee5a0b546004cae491639ecf24fe0b9ef4000a06802b4f6dd0ff7d9e1202aaa1"} Oct 30 03:21:01.701410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.701268 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:01.702522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.702500 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:01.811537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.811508 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:01.846798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.846738 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:21:01.902449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902351 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-data\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.902449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902426 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-istio-token\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-istio-token\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.902677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902475 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-socket\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.902677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902524 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d818414-c583-416a-9561-927249e337e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d818414-c583-416a-9561-927249e337e5-istio-podinfo\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.902677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902577 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-envoy\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.902677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902649 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-kube-api-access-4d76s\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-kube-api-access-4d76s\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.902953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902679 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-certs\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.902953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902710 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-credential-socket\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.903051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.902951 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d818414-c583-416a-9561-927249e337e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d818414-c583-416a-9561-927249e337e5-istiod-ca-cert\") pod \"9d818414-c583-416a-9561-927249e337e5\" (UID: \"9d818414-c583-416a-9561-927249e337e5\") " Oct 30 03:21:01.903227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.903195 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-socket" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:01.903368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.903341 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-certs" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:01.903499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.903472 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-credential-socket" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:01.903738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.903536 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:01.903738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.903559 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:01.903738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.903572 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d818414-c583-416a-9561-927249e337e5-istiod-ca-cert" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:01.903738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.903629 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-data" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:01.905558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.905524 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-envoy" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:01.906086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.906058 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-kube-api-access-4d76s" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "kube-api-access-4d76s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:01.906202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.906124 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d818414-c583-416a-9561-927249e337e5-istio-podinfo" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:21:01.906569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:01.906550 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-istio-token" pod "9d818414-c583-416a-9561-927249e337e5" (UID: "9d818414-c583-416a-9561-927249e337e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:02.004396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.004364 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d818414-c583-416a-9561-927249e337e5-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:02.004396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.004393 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:02.004396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.004401 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:02.004643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.004410 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d818414-c583-416a-9561-927249e337e5-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:02.004643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.004423 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:02.004643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.004435 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d76s\" (UniqueName: \"kubernetes.io/projected/9d818414-c583-416a-9561-927249e337e5-kube-api-access-4d76s\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:02.004643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.004444 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d818414-c583-416a-9561-927249e337e5-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:02.708706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.708673 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" Oct 30 03:21:02.709165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.708722 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql" event={"ID":"9d818414-c583-416a-9561-927249e337e5","Type":"ContainerDied","Data":"5ae0f8dc03764197831dc7d457c12dfd9a6074c9cae5385a82c122f5aebd6895"} Oct 30 03:21:02.709165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.708813 4442 scope.go:122] "RemoveContainer" containerID="ee5a0b546004cae491639ecf24fe0b9ef4000a06802b4f6dd0ff7d9e1202aaa1" Oct 30 03:21:02.779748 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.779712 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql"] Oct 30 03:21:02.791196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:02.791175 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-n7lql"] Oct 30 03:21:03.085989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:03.085963 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:03.203839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:03.203799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:03.394185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:03.394102 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:04.106267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.106233 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d818414-c583-416a-9561-927249e337e5" path="/var/lib/kubelet/pods/9d818414-c583-416a-9561-927249e337e5/volumes" Oct 30 03:21:04.556467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.556445 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:21:04.632186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632155 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-socket\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.632326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632223 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istiod-ca-cert\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.632326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632267 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-envoy\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.632326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632292 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-data\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.632326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632313 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-podinfo\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.632540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632339 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-token\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.632540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632385 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-kube-api-access-7xbdl\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-kube-api-access-7xbdl\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.632540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632455 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-socket" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:04.632690 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632626 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-data" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:04.632800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632734 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istiod-ca-cert" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:04.632942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632923 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.633015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632945 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.633015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.632959 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.634745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.634714 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-envoy" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:04.634883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.634793 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-token" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:04.635159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.635134 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-kube-api-access-7xbdl" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "kube-api-access-7xbdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:04.635257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.635227 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-podinfo" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:21:04.723312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.723232 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerID="53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00" exitCode=0 Oct 30 03:21:04.723312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.723268 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" event={"ID":"9e3313fb-3e34-42bc-a4b5-aa52df7756ec","Type":"ContainerDied","Data":"53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00"} Oct 30 03:21:04.723312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.723282 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" Oct 30 03:21:04.723312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.723310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq" event={"ID":"9e3313fb-3e34-42bc-a4b5-aa52df7756ec","Type":"ContainerDied","Data":"0f517bbbae60b12c3a7e23dd9b72b506f4d963843a33ef72a6e871bf4185173e"} Oct 30 03:21:04.723567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.723330 4442 scope.go:122] "RemoveContainer" containerID="53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00" Oct 30 03:21:04.733273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733253 4442 scope.go:122] "RemoveContainer" containerID="3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e" Oct 30 03:21:04.733554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733529 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-certs\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.733635 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733577 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-credential-socket\") pod \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\" (UID: \"9e3313fb-3e34-42bc-a4b5-aa52df7756ec\") " Oct 30 03:21:04.733874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733842 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-credential-socket" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:04.733874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733851 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-certs" pod "9e3313fb-3e34-42bc-a4b5-aa52df7756ec" (UID: "9e3313fb-3e34-42bc-a4b5-aa52df7756ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:04.734003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733960 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.734003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733974 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.734003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733983 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.734003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.733992 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbdl\" (UniqueName: \"kubernetes.io/projected/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-kube-api-access-7xbdl\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.734003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.734001 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.734242 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.734011 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3313fb-3e34-42bc-a4b5-aa52df7756ec-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:04.741971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.741954 4442 scope.go:122] "RemoveContainer" containerID="53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00" Oct 30 03:21:04.742236 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:04.742214 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00\": container with ID starting with 53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00 not found: ID does not exist" containerID="53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00" Oct 30 03:21:04.742306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.742248 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00"} err="failed to get container status \"53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00\": rpc error: code = NotFound desc = could not find container \"53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00\": container with ID starting with 53faddf1169d27988251b98c894c554265c3aea239ea3f0723d0f9e2e0d2fc00 not found: ID does not exist" Oct 30 03:21:04.742306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.742273 4442 scope.go:122] "RemoveContainer" containerID="3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e" Oct 30 03:21:04.742544 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:04.742523 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e\": container with ID starting with 3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e not found: ID does not exist" containerID="3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e" Oct 30 03:21:04.742587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.742551 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e"} err="failed to get container status \"3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e\": rpc error: code = NotFound desc = could not find container \"3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e\": container with ID starting with 3ad93721f5662f022bae961489036bca5aa8375221ff76f481d0e0000cffed1e not found: ID does not exist" Oct 30 03:21:04.837497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.837462 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:04.937185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:04.937142 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:05.070125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:05.070098 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq"] Oct 30 03:21:05.073404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:05.073382 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-585938-openshift-default-7d8b79888f-t8kxq"] Oct 30 03:21:05.246999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:05.246960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:05.945482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:05.945438 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr"] Oct 30 03:21:06.075382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:06.075342 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:06.106941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:06.106913 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" path="/var/lib/kubelet/pods/9e3313fb-3e34-42bc-a4b5-aa52df7756ec/volumes" Oct 30 03:21:06.211834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:06.211723 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:06.776031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:06.775999 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt"] Oct 30 03:21:06.776475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:06.776207 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" podUID="6a09fae0-0791-42f7-900a-a7439869821a" containerName="maas-api" containerID="cri-o://e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95" gracePeriod=30 Oct 30 03:21:06.850966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:06.850941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:07.008231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.008204 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:07.100857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.100010 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:21:07.193393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.193364 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-metrics-certs\") pod \"6a09fae0-0791-42f7-900a-a7439869821a\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " Oct 30 03:21:07.193577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.193401 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-maas-api-tls\") pod \"6a09fae0-0791-42f7-900a-a7439869821a\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " Oct 30 03:21:07.193577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.193512 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a09fae0-0791-42f7-900a-a7439869821a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a09fae0-0791-42f7-900a-a7439869821a-openshift-service-ca\") pod \"6a09fae0-0791-42f7-900a-a7439869821a\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " Oct 30 03:21:07.193577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.193548 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a09fae0-0791-42f7-900a-a7439869821a-kube-api-access-ngqdr\" (UniqueName: \"kubernetes.io/projected/6a09fae0-0791-42f7-900a-a7439869821a-kube-api-access-ngqdr\") pod \"6a09fae0-0791-42f7-900a-a7439869821a\" (UID: \"6a09fae0-0791-42f7-900a-a7439869821a\") " Oct 30 03:21:07.194008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.193949 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a09fae0-0791-42f7-900a-a7439869821a-openshift-service-ca" pod "6a09fae0-0791-42f7-900a-a7439869821a" (UID: "6a09fae0-0791-42f7-900a-a7439869821a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:07.194111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.194061 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a09fae0-0791-42f7-900a-a7439869821a-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:07.195872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.195845 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-maas-api-tls" pod "6a09fae0-0791-42f7-900a-a7439869821a" (UID: "6a09fae0-0791-42f7-900a-a7439869821a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:21:07.195965 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.195893 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-metrics-certs" pod "6a09fae0-0791-42f7-900a-a7439869821a" (UID: "6a09fae0-0791-42f7-900a-a7439869821a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:21:07.195965 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.195907 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a09fae0-0791-42f7-900a-a7439869821a-kube-api-access-ngqdr" pod "6a09fae0-0791-42f7-900a-a7439869821a" (UID: "6a09fae0-0791-42f7-900a-a7439869821a"). InnerVolumeSpecName "kube-api-access-ngqdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:07.334439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.334352 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:07.334439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.334394 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a09fae0-0791-42f7-900a-a7439869821a-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:07.334439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.334411 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngqdr\" (UniqueName: \"kubernetes.io/projected/6a09fae0-0791-42f7-900a-a7439869821a-kube-api-access-ngqdr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:07.506514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.506485 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n"] Oct 30 03:21:07.506947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.506935 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" Oct 30 03:21:07.506989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.506948 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" Oct 30 03:21:07.506989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.506973 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" Oct 30 03:21:07.506989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.506979 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" Oct 30 03:21:07.507097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.506992 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6a09fae0-0791-42f7-900a-a7439869821a" containerName="maas-api" Oct 30 03:21:07.507097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.506997 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a09fae0-0791-42f7-900a-a7439869821a" containerName="maas-api" Oct 30 03:21:07.507097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.507065 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" Oct 30 03:21:07.507097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.507074 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" Oct 30 03:21:07.507097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.507083 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d818414-c583-416a-9561-927249e337e5" containerName="istio-proxy" Oct 30 03:21:07.507097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.507091 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a09fae0-0791-42f7-900a-a7439869821a" containerName="maas-api" Oct 30 03:21:07.510388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.510372 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.527124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.527103 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b132360f-serving-cert\"" Oct 30 03:21:07.530593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.530568 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n"] Oct 30 03:21:07.536188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.536161 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.536327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.536206 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ba7153-5eb1-4254-8b4b-81daaf6ae865-openshift-service-ca\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.536327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.536236 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29lt\" (UniqueName: \"kubernetes.io/projected/38ba7153-5eb1-4254-8b4b-81daaf6ae865-kube-api-access-d29lt\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.536327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.536297 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-metrics-certs\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.603051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.603020 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:07.636945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.636919 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.637106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.636948 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ba7153-5eb1-4254-8b4b-81daaf6ae865-openshift-service-ca\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.637106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.636967 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29lt\" (UniqueName: \"kubernetes.io/projected/38ba7153-5eb1-4254-8b4b-81daaf6ae865-kube-api-access-d29lt\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.637106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.637006 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-metrics-certs\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.637106 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:07.637075 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-serving-cert: secret "maas-api-e2e-mt-b132360f-serving-cert" not found Oct 30 03:21:07.637325 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:07.637161 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls podName:38ba7153-5eb1-4254-8b4b-81daaf6ae865 nodeName:}" failed. No retries permitted until 2026-10-30 03:21:08.137139104 +0000 UTC m=+1092.139492299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls") pod "maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" (UID: "38ba7153-5eb1-4254-8b4b-81daaf6ae865") : secret "maas-api-e2e-mt-b132360f-serving-cert" not found Oct 30 03:21:07.637585 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.637560 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ba7153-5eb1-4254-8b4b-81daaf6ae865-openshift-service-ca\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.639544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.639525 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-metrics-certs\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.655506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.655487 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29lt\" (UniqueName: \"kubernetes.io/projected/38ba7153-5eb1-4254-8b4b-81daaf6ae865-kube-api-access-d29lt\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:07.704691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.704666 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:07.742702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.742676 4442 generic.go:356] "Generic (PLEG): container finished" podID="6a09fae0-0791-42f7-900a-a7439869821a" containerID="e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95" exitCode=0 Oct 30 03:21:07.742818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.742723 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" Oct 30 03:21:07.742818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.742786 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" event={"ID":"6a09fae0-0791-42f7-900a-a7439869821a","Type":"ContainerDied","Data":"e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95"} Oct 30 03:21:07.742910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.742841 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt" event={"ID":"6a09fae0-0791-42f7-900a-a7439869821a","Type":"ContainerDied","Data":"d62308febda5bbfb686841183d7204456dd76ab366334dabd441520d3570b5fb"} Oct 30 03:21:07.742910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.742864 4442 scope.go:122] "RemoveContainer" containerID="e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95" Oct 30 03:21:07.752675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.752660 4442 scope.go:122] "RemoveContainer" containerID="e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95" Oct 30 03:21:07.752923 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:07.752907 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95\": container with ID starting with e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95 not found: ID does not exist" containerID="e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95" Oct 30 03:21:07.752988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.752934 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95"} err="failed to get container status \"e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95\": rpc error: code = NotFound desc = could not find container \"e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95\": container with ID starting with e762089572066562c37e719f61086ca6d198fcc5f392ddbbd31ec89f2eb8fb95 not found: ID does not exist" Oct 30 03:21:07.804047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.803945 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt"] Oct 30 03:21:07.823521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.817834 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3e4daf-59d6868d48-4jqnt"] Oct 30 03:21:07.824608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:07.824391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:08.107131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:08.107060 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a09fae0-0791-42f7-900a-a7439869821a" path="/var/lib/kubelet/pods/6a09fae0-0791-42f7-900a-a7439869821a/volumes" Oct 30 03:21:08.142730 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:08.142701 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:08.145444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:08.145421 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls\") pod \"maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:08.424124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:08.424049 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:08.754788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:08.752471 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n"] Oct 30 03:21:08.889118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:08.889090 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:08.992524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:08.992501 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:09.415729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:09.415696 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:09.762726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:09.762697 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:09.769006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:09.768978 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" event={"ID":"38ba7153-5eb1-4254-8b4b-81daaf6ae865","Type":"ContainerStarted","Data":"1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107"} Oct 30 03:21:09.769123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:09.769031 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" event={"ID":"38ba7153-5eb1-4254-8b4b-81daaf6ae865","Type":"ContainerStarted","Data":"b960c514f76f217ee8179ff99ab8f34e13c46460fd7d9a09c12fbdfceaa406ab"} Oct 30 03:21:09.769193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:09.769177 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:09.815041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:09.814899 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" podStartSLOduration=2.382160193 podStartE2EDuration="2.814882981s" totalImagesPullingTime="432.722788ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:09.814875518 +0000 UTC m=+1093.817228726" watchObservedRunningTime="2026-10-30 03:21:09.814882981 +0000 UTC m=+1093.817236184" Oct 30 03:21:10.160334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:10.160218 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:10.946141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:10.946091 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" podUID="89212ab1-68bb-4e18-a406-a997324528ec" containerName="payload-processing" containerID="cri-o://f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70" gracePeriod=25 Oct 30 03:21:11.208736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.208665 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:21:11.291661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.291628 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89212ab1-68bb-4e18-a406-a997324528ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89212ab1-68bb-4e18-a406-a997324528ec-plugins-config-volume\") pod \"89212ab1-68bb-4e18-a406-a997324528ec\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " Oct 30 03:21:11.291849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.291697 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89212ab1-68bb-4e18-a406-a997324528ec-kube-api-access-l7cxz\" (UniqueName: \"kubernetes.io/projected/89212ab1-68bb-4e18-a406-a997324528ec-kube-api-access-l7cxz\") pod \"89212ab1-68bb-4e18-a406-a997324528ec\" (UID: \"89212ab1-68bb-4e18-a406-a997324528ec\") " Oct 30 03:21:11.292103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.292077 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89212ab1-68bb-4e18-a406-a997324528ec-plugins-config-volume" pod "89212ab1-68bb-4e18-a406-a997324528ec" (UID: "89212ab1-68bb-4e18-a406-a997324528ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:11.292217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.292203 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89212ab1-68bb-4e18-a406-a997324528ec-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:11.293975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.293954 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89212ab1-68bb-4e18-a406-a997324528ec-kube-api-access-l7cxz" pod "89212ab1-68bb-4e18-a406-a997324528ec" (UID: "89212ab1-68bb-4e18-a406-a997324528ec"). InnerVolumeSpecName "kube-api-access-l7cxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:11.393799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.393737 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7cxz\" (UniqueName: \"kubernetes.io/projected/89212ab1-68bb-4e18-a406-a997324528ec-kube-api-access-l7cxz\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:11.490465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.490379 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:11.789653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.789621 4442 generic.go:356] "Generic (PLEG): container finished" podID="89212ab1-68bb-4e18-a406-a997324528ec" containerID="f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70" exitCode=0 Oct 30 03:21:11.789866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.789687 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" Oct 30 03:21:11.789866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.789694 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" event={"ID":"89212ab1-68bb-4e18-a406-a997324528ec","Type":"ContainerDied","Data":"f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70"} Oct 30 03:21:11.789866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.789791 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr" event={"ID":"89212ab1-68bb-4e18-a406-a997324528ec","Type":"ContainerDied","Data":"dc11ec9d201c222d85fb0e8c23a8a0e50431ce4b50cc05fc05e34ecf282db32c"} Oct 30 03:21:11.789866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.789815 4442 scope.go:122] "RemoveContainer" containerID="f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70" Oct 30 03:21:11.801707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.801690 4442 scope.go:122] "RemoveContainer" containerID="f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70" Oct 30 03:21:11.802022 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:11.802003 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70\": container with ID starting with f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70 not found: ID does not exist" containerID="f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70" Oct 30 03:21:11.802096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.802030 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70"} err="failed to get container status \"f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70\": rpc error: code = NotFound desc = could not find container \"f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70\": container with ID starting with f5ef4d67bd73effdeea4bdba735bc35d3b2b645371f0ceed874985ec3351ce70 not found: ID does not exist" Oct 30 03:21:11.836675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.836649 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr"] Oct 30 03:21:11.847709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:11.847689 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cc0859-74dbb467c6-qvffr"] Oct 30 03:21:12.106988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:12.106906 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89212ab1-68bb-4e18-a406-a997324528ec" path="/var/lib/kubelet/pods/89212ab1-68bb-4e18-a406-a997324528ec/volumes" Oct 30 03:21:13.439553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:13.439523 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:13.563485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:13.563456 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:15.126697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:15.126655 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:15.779697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:15.779666 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:21:16.608617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.608582 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r"] Oct 30 03:21:16.609138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.608824 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" podUID="8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" containerName="maas-api" containerID="cri-o://ffd485b9d213fd924b549b1712fd0ab759636ff526560de5391562c84c321aed" gracePeriod=30 Oct 30 03:21:16.676146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.676111 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:16.819110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.819077 4442 generic.go:356] "Generic (PLEG): container finished" podID="8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" containerID="ffd485b9d213fd924b549b1712fd0ab759636ff526560de5391562c84c321aed" exitCode=0 Oct 30 03:21:16.819269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.819147 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" event={"ID":"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9","Type":"ContainerDied","Data":"ffd485b9d213fd924b549b1712fd0ab759636ff526560de5391562c84c321aed"} Oct 30 03:21:16.855144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.855121 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:21:16.953319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.953234 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-kube-api-access-r56zg\" (UniqueName: \"kubernetes.io/projected/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-kube-api-access-r56zg\") pod \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " Oct 30 03:21:16.953480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.953341 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-openshift-service-ca\") pod \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " Oct 30 03:21:16.953480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.953464 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls\") pod \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " Oct 30 03:21:16.953557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.953502 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-metrics-certs\") pod \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\" (UID: \"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9\") " Oct 30 03:21:16.954024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.953989 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-openshift-service-ca" pod "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" (UID: "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:16.956121 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.956075 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-kube-api-access-r56zg" pod "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" (UID: "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9"). InnerVolumeSpecName "kube-api-access-r56zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:16.956470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.956438 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls" pod "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" (UID: "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:21:16.956576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:16.956552 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-metrics-certs" pod "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" (UID: "8acd754c-5d8b-4a1a-9efb-68448dbb7fa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:21:17.104777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.087453 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:17.104777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.087487 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:17.104777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.087501 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56zg\" (UniqueName: \"kubernetes.io/projected/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-kube-api-access-r56zg\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:17.104777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.087517 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:17.462727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.462691 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:17.826087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.826057 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" Oct 30 03:21:17.826503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.826085 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r" event={"ID":"8acd754c-5d8b-4a1a-9efb-68448dbb7fa9","Type":"ContainerDied","Data":"70fb7764b8097d59ac47453e9bf34b65334cf4dd1dbc3d004f24ea4679350629"} Oct 30 03:21:17.826503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.826147 4442 scope.go:122] "RemoveContainer" containerID="ffd485b9d213fd924b549b1712fd0ab759636ff526560de5391562c84c321aed" Oct 30 03:21:17.872891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.872861 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r"] Oct 30 03:21:17.883340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:17.883316 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cc0859-67976b5759-bbt4r"] Oct 30 03:21:18.106573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.106494 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" path="/var/lib/kubelet/pods/8acd754c-5d8b-4a1a-9efb-68448dbb7fa9/volumes" Oct 30 03:21:18.966962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.966934 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw"] Oct 30 03:21:18.967565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967396 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" containerName="maas-api" Oct 30 03:21:18.967565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967412 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" containerName="maas-api" Oct 30 03:21:18.967565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967427 4442 cpu_manager.go:414] "containerMap: removing container" podUID="89212ab1-68bb-4e18-a406-a997324528ec" containerName="payload-processing" Oct 30 03:21:18.967565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967431 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89212ab1-68bb-4e18-a406-a997324528ec" containerName="payload-processing" Oct 30 03:21:18.967565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967459 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" Oct 30 03:21:18.967565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967468 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3313fb-3e34-42bc-a4b5-aa52df7756ec" containerName="istio-proxy" Oct 30 03:21:18.967870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967613 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acd754c-5d8b-4a1a-9efb-68448dbb7fa9" containerName="maas-api" Oct 30 03:21:18.967870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.967627 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="89212ab1-68bb-4e18-a406-a997324528ec" containerName="payload-processing" Oct 30 03:21:18.972131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.972113 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:18.985596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:18.985570 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d4b0d5-serving-cert\"" Oct 30 03:21:19.000271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.000244 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw"] Oct 30 03:21:19.051770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.051734 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s"] Oct 30 03:21:19.055636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.055617 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.059736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.059698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:19.075519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.075486 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s"] Oct 30 03:21:19.116681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.116649 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspk9\" (UniqueName: \"kubernetes.io/projected/048fe785-48cf-4594-ab8c-a2b65cc87eeb-kube-api-access-sspk9\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.116882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.116698 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048fe785-48cf-4594-ab8c-a2b65cc87eeb-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.116955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.116909 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-maas-api-tls\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.117037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.117018 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-metrics-certs\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspk9\" (UniqueName: \"kubernetes.io/projected/048fe785-48cf-4594-ab8c-a2b65cc87eeb-kube-api-access-sspk9\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221673 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048fe785-48cf-4594-ab8c-a2b65cc87eeb-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221723 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221835 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-maas-api-tls\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221873 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221915 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea287de4-a030-4e5e-b106-451729ab7806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221948 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221974 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.221999 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea287de4-a030-4e5e-b106-451729ab7806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.222066 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.222097 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c468\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-kube-api-access-6c468\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.222296 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-metrics-certs\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.222369 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.223460 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048fe785-48cf-4594-ab8c-a2b65cc87eeb-openshift-service-ca\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.231825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.230929 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-metrics-certs\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.242216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.242113 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspk9\" (UniqueName: \"kubernetes.io/projected/048fe785-48cf-4594-ab8c-a2b65cc87eeb-kube-api-access-sspk9\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.242554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.242528 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-maas-api-tls\") pod \"maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.284883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.284857 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:19.323727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.323696 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.323918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.323811 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.323918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.323887 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.323922 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea287de4-a030-4e5e-b106-451729ab7806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.323952 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.323981 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324006 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea287de4-a030-4e5e-b106-451729ab7806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324064 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324091 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c468\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-kube-api-access-6c468\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324174 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324261 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324531 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324619 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.324818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.324778 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea287de4-a030-4e5e-b106-451729ab7806-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.327251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.326599 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.327251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.326946 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea287de4-a030-4e5e-b106-451729ab7806-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.343024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.342939 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c468\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-kube-api-access-6c468\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.343024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.342997 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-h6l2s\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.366889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.366863 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:19.372906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.371284 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:19.583210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.583182 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw"] Oct 30 03:21:19.584678 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:21:19.584636 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048fe785_48cf_4594_ab8c_a2b65cc87eeb.slice/crio-5b44643cd1ea4360c4bb9102d7356fd54475dd90c2578f284a0f3a21e7686ecb WatchSource:0}: Error finding container 5b44643cd1ea4360c4bb9102d7356fd54475dd90c2578f284a0f3a21e7686ecb: Status 404 returned error can't find the container with id 5b44643cd1ea4360c4bb9102d7356fd54475dd90c2578f284a0f3a21e7686ecb Oct 30 03:21:19.679797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.679740 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:19.686590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.686567 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s"] Oct 30 03:21:19.687785 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:21:19.687747 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea287de4_a030_4e5e_b106_451729ab7806.slice/crio-56e0ba5475a5c057ae1b2f4cd9ef03a09327cea7cef22964cfa85a114f85a8cc WatchSource:0}: Error finding container 56e0ba5475a5c057ae1b2f4cd9ef03a09327cea7cef22964cfa85a114f85a8cc: Status 404 returned error can't find the container with id 56e0ba5475a5c057ae1b2f4cd9ef03a09327cea7cef22964cfa85a114f85a8cc Oct 30 03:21:19.690093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.690064 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:21:19.690174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.690125 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:21:19.690174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.690152 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:21:19.780607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.780512 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:19.839478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.839380 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" event={"ID":"048fe785-48cf-4594-ab8c-a2b65cc87eeb","Type":"ContainerStarted","Data":"5b44643cd1ea4360c4bb9102d7356fd54475dd90c2578f284a0f3a21e7686ecb"} Oct 30 03:21:19.841402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.841375 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" event={"ID":"ea287de4-a030-4e5e-b106-451729ab7806","Type":"ContainerStarted","Data":"2e664f97e6322208d7d1b808225ee821ff22f368b03ddc65926e257079df58ad"} Oct 30 03:21:19.841502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.841425 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" event={"ID":"ea287de4-a030-4e5e-b106-451729ab7806","Type":"ContainerStarted","Data":"56e0ba5475a5c057ae1b2f4cd9ef03a09327cea7cef22964cfa85a114f85a8cc"} Oct 30 03:21:19.884612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:19.884554 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podStartSLOduration=0.884535725 podStartE2EDuration="884.535725ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:19.882090862 +0000 UTC m=+1103.884444059" watchObservedRunningTime="2026-10-30 03:21:19.884535725 +0000 UTC m=+1103.886888924" Oct 30 03:21:20.372145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.372105 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:20.374326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.374297 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:20.374475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.374349 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:20.443070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.443024 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:20.847993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.847952 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" event={"ID":"048fe785-48cf-4594-ab8c-a2b65cc87eeb","Type":"ContainerStarted","Data":"5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432"} Oct 30 03:21:20.848198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.848181 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:20.857840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.857812 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:20.897776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.891317 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" podStartSLOduration=2.507878602 podStartE2EDuration="2.891294393s" totalImagesPullingTime="383.415791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:20.890200418 +0000 UTC m=+1104.892553616" watchObservedRunningTime="2026-10-30 03:21:20.891294393 +0000 UTC m=+1104.893647593" Oct 30 03:21:20.993790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:20.988136 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:21.372009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:21.371973 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:21.372184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:21.372044 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:22.371866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:22.371828 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:22.372048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:22.371882 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:22.925021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:22.924861 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:23.371885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:23.371853 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:23.372037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:23.371901 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:24.372304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:24.372263 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:24.372781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:24.372333 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:24.642505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:24.642423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:24.861319 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:24.861284 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:21:25.372112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:25.372079 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:25.372289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:25.372136 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:25.514827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:25.513272 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:21:26.372726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:26.372687 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:26.372936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:26.372784 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:26.516214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:26.516180 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:26.859911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:26.859885 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:21:27.372656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:27.372622 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 03:21:27.372862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:27.372673 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 03:21:28.362864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:28.362826 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:28.617795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:28.617678 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:28.618184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:28.618158 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:28.619436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:28.619415 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:21:30.202025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:30.201989 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:31.790933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:31.790902 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:32.803273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:32.803244 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt"] Oct 30 03:21:32.803725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:32.803428 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" containerID="cri-o://ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477" gracePeriod=30 Oct 30 03:21:32.958571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:32.958540 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:33.494031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:33.493995 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:33.641041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:33.641007 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:33.751814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:33.751783 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:35.272011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:35.271979 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:36.926830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:36.925980 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:37.039119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:37.039077 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:37.522156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:37.522079 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:38.907696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:38.888112 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:39.029778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.029629 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:21:39.050352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.049221 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:39.116106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.116072 4442 generic.go:356] "Generic (PLEG): container finished" podID="18e14c69-96b5-4171-8684-64fa945d7d96" containerID="ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477" exitCode=0 Oct 30 03:21:39.116407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.116382 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" event={"ID":"18e14c69-96b5-4171-8684-64fa945d7d96","Type":"ContainerDied","Data":"ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477"} Oct 30 03:21:39.116577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.116560 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" event={"ID":"18e14c69-96b5-4171-8684-64fa945d7d96","Type":"ContainerDied","Data":"f2be136df220b0f2daee80345d74f7fa46be0abeb4a1f18c6588e3831e4fc301"} Oct 30 03:21:39.116696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.116683 4442 scope.go:122] "RemoveContainer" containerID="ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477" Oct 30 03:21:39.116996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.116977 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt" Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.225984 4442 scope.go:122] "RemoveContainer" containerID="ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477" Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:39.226472 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477\": container with ID starting with ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477 not found: ID does not exist" containerID="ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477" Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.226517 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477"} err="failed to get container status \"ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477\": rpc error: code = NotFound desc = could not find container \"ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477\": container with ID starting with ff8f8db4f6a30aabffa2bda870d74c83f4bc533a21e6ce7917f062e3251f3477 not found: ID does not exist" Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227240 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-credential-socket\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-certs\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227333 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-socket\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227403 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-envoy\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227453 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e14c69-96b5-4171-8684-64fa945d7d96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e14c69-96b5-4171-8684-64fa945d7d96-istio-podinfo\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227492 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e14c69-96b5-4171-8684-64fa945d7d96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e14c69-96b5-4171-8684-64fa945d7d96-istiod-ca-cert\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227531 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-data\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227565 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-kube-api-access-wlct6\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-kube-api-access-wlct6\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.227670 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-istio-token\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-istio-token\") pod \"18e14c69-96b5-4171-8684-64fa945d7d96\" (UID: \"18e14c69-96b5-4171-8684-64fa945d7d96\") " Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.234076 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-certs" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:39.240779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.234291 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-socket" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:39.245905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.245877 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-data" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:39.273953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.251015 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e14c69-96b5-4171-8684-64fa945d7d96-istiod-ca-cert" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:39.273953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.251078 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-credential-socket" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.336814 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.336845 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.336859 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e14c69-96b5-4171-8684-64fa945d7d96-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.336872 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.336884 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.337326 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-istio-token" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.337449 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e14c69-96b5-4171-8684-64fa945d7d96-istio-podinfo" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:21:39.344991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.337512 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-envoy" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:39.352775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.351879 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-kube-api-access-wlct6" pod "18e14c69-96b5-4171-8684-64fa945d7d96" (UID: "18e14c69-96b5-4171-8684-64fa945d7d96"). InnerVolumeSpecName "kube-api-access-wlct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:39.440746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.440653 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.440746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.440690 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e14c69-96b5-4171-8684-64fa945d7d96-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.440746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.440704 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e14c69-96b5-4171-8684-64fa945d7d96-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.440746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.440718 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlct6\" (UniqueName: \"kubernetes.io/projected/18e14c69-96b5-4171-8684-64fa945d7d96-kube-api-access-wlct6\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:21:39.631077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.630744 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt"] Oct 30 03:21:39.662784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.662554 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e44313-openshift-default-d5f88445f-qjxzt"] Oct 30 03:21:39.907910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:39.907616 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:40.007868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.004306 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:40.120710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.120680 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:40.125550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.125515 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" path="/var/lib/kubelet/pods/18e14c69-96b5-4171-8684-64fa945d7d96/volumes" Oct 30 03:21:40.136672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.136644 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss"] Oct 30 03:21:40.137131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.137116 4442 cpu_manager.go:414] "containerMap: removing container" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" Oct 30 03:21:40.137179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.137132 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" Oct 30 03:21:40.137223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.137215 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e14c69-96b5-4171-8684-64fa945d7d96" containerName="istio-proxy" Oct 30 03:21:40.141825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.141804 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.159952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.159900 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d6c536-dockercfg-4wmlr\"" Oct 30 03:21:40.162612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.162580 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss"] Oct 30 03:21:40.172365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.172349 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d6c536\"" Oct 30 03:21:40.180564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.180535 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgn7\" (UniqueName: \"kubernetes.io/projected/9778eefd-ec5d-4ed8-898c-6dd2265544d7-kube-api-access-jfgn7\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.180877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.180855 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9778eefd-ec5d-4ed8-898c-6dd2265544d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.330212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.330133 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9778eefd-ec5d-4ed8-898c-6dd2265544d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.330622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.330507 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9778eefd-ec5d-4ed8-898c-6dd2265544d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.330973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.330950 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgn7\" (UniqueName: \"kubernetes.io/projected/9778eefd-ec5d-4ed8-898c-6dd2265544d7-kube-api-access-jfgn7\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.337201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.337074 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:40.360470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.360440 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgn7\" (UniqueName: \"kubernetes.io/projected/9778eefd-ec5d-4ed8-898c-6dd2265544d7-kube-api-access-jfgn7\") pod \"payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.455382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.455290 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:40.774094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:40.774067 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss"] Oct 30 03:21:40.775930 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:21:40.775898 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9778eefd_ec5d_4ed8_898c_6dd2265544d7.slice/crio-54efc9253c9226ee34af6a510d638502b31d5518c1994e5c3fd5260983b424d1 WatchSource:0}: Error finding container 54efc9253c9226ee34af6a510d638502b31d5518c1994e5c3fd5260983b424d1: Status 404 returned error can't find the container with id 54efc9253c9226ee34af6a510d638502b31d5518c1994e5c3fd5260983b424d1 Oct 30 03:21:41.131017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:41.130919 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" event={"ID":"9778eefd-ec5d-4ed8-898c-6dd2265544d7","Type":"ContainerStarted","Data":"9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4"} Oct 30 03:21:41.131419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:41.131015 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" event={"ID":"9778eefd-ec5d-4ed8-898c-6dd2265544d7","Type":"ContainerStarted","Data":"54efc9253c9226ee34af6a510d638502b31d5518c1994e5c3fd5260983b424d1"} Oct 30 03:21:41.131419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:41.131045 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:41.168076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:41.168013 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" podStartSLOduration=1.167996029 podStartE2EDuration="1.167996029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:41.166745349 +0000 UTC m=+1125.169098548" watchObservedRunningTime="2026-10-30 03:21:41.167996029 +0000 UTC m=+1125.170349230" Oct 30 03:21:41.832529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:41.832491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:43.525803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:43.525764 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:44.176223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:44.176177 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:45.214662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:45.214629 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:46.162835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.162797 4442 generic.go:356] "Generic (PLEG): container finished" podID="83237762-f446-40ca-9500-861aaf8a326e" containerID="49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b" exitCode=0 Oct 30 03:21:46.163106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.162846 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" event={"ID":"83237762-f446-40ca-9500-861aaf8a326e","Type":"ContainerDied","Data":"49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b"} Oct 30 03:21:46.163289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.163269 4442 scope.go:122] "RemoveContainer" containerID="49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b" Oct 30 03:21:46.164677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.164643 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:21:46.164809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.164732 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:21:46.164809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.164800 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:21:46.548668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.548641 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:46.701584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.701544 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:46.729735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.729698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:46.838438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:46.838340 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:47.072412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:47.072384 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:47.194817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:47.194703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" event={"ID":"83237762-f446-40ca-9500-861aaf8a326e","Type":"ContainerStarted","Data":"7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8"} Oct 30 03:21:47.618258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:47.618218 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:47.619529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:47.619508 4442 patch_prober.go:29] interesting pod/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:21:47.619602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:47.619554 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 03:21:48.618211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:48.618179 4442 patch_prober.go:29] interesting pod/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:21:48.618392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:48.618234 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 03:21:48.808218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:48.808169 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:21:48.955214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:48.955126 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:49.618280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:49.618237 4442 patch_prober.go:29] interesting pod/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:21:49.618543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:49.618516 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 03:21:50.619160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:50.619102 4442 patch_prober.go:29] interesting pod/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:21:50.619666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:50.619183 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 03:21:50.761494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:50.761461 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:51.618911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:51.618871 4442 patch_prober.go:29] interesting pod/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 03:21:51.619117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:51.618936 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 03:21:52.140624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:52.140592 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:21:52.561080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:52.561048 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:52.655123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:52.655090 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:52.656373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:52.656348 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:52.658145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:52.658125 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:21:52.686305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:52.686279 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:54.109459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.109428 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6"] Oct 30 03:21:54.114612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.114588 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.141094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.141065 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d6c536-serving-cert\"" Oct 30 03:21:54.144808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.144780 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6"] Oct 30 03:21:54.191930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.191903 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:54.252260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.252233 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.252260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.252263 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mxn2\" (UniqueName: \"kubernetes.io/projected/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-kube-api-access-2mxn2\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.252471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.252431 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.252471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.252459 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-metrics-certs\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.315783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.315726 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:54.353873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.353825 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-metrics-certs\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.354050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.354019 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.354050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.354045 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mxn2\" (UniqueName: \"kubernetes.io/projected/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-kube-api-access-2mxn2\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.354232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.354214 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.354312 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:54.354297 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-serving-cert: secret "maas-api-e2e-ipp-a-d6c536-serving-cert" not found Oct 30 03:21:54.354396 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:54.354375 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls podName:1ae1db7c-ff63-4348-b529-18f27b7a5ccb nodeName:}" failed. No retries permitted until 2026-10-30 03:21:54.854347773 +0000 UTC m=+1138.856700948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls") pod "maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" (UID: "1ae1db7c-ff63-4348-b529-18f27b7a5ccb") : secret "maas-api-e2e-ipp-a-d6c536-serving-cert" not found Oct 30 03:21:54.354842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.354821 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.356578 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.356560 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-metrics-certs\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.374804 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.374728 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mxn2\" (UniqueName: \"kubernetes.io/projected/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-kube-api-access-2mxn2\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.513341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.513312 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:54.858527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.858493 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:54.860479 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:21:54.860445 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:21:54.861048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:54.861030 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:55.027656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:55.027626 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:55.325664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:55.325641 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6"] Oct 30 03:21:55.326949 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:21:55.326903 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae1db7c_ff63_4348_b529_18f27b7a5ccb.slice/crio-72940fe0dcd42c173a62b711fc04ca0694547189057f9cfee8c362882f1e3779 WatchSource:0}: Error finding container 72940fe0dcd42c173a62b711fc04ca0694547189057f9cfee8c362882f1e3779: Status 404 returned error can't find the container with id 72940fe0dcd42c173a62b711fc04ca0694547189057f9cfee8c362882f1e3779 Oct 30 03:21:55.929489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:55.929459 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:56.257824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:56.257784 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" event={"ID":"1ae1db7c-ff63-4348-b529-18f27b7a5ccb","Type":"ContainerStarted","Data":"9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d"} Oct 30 03:21:56.257972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:56.257860 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" event={"ID":"1ae1db7c-ff63-4348-b529-18f27b7a5ccb","Type":"ContainerStarted","Data":"72940fe0dcd42c173a62b711fc04ca0694547189057f9cfee8c362882f1e3779"} Oct 30 03:21:56.257972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:56.257893 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:21:56.296717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:56.296663 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" podStartSLOduration=1.922774258 podStartE2EDuration="2.296650622s" totalImagesPullingTime="373.876364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:21:56.29399772 +0000 UTC m=+1140.296350916" watchObservedRunningTime="2026-10-30 03:21:56.296650622 +0000 UTC m=+1140.299003818" Oct 30 03:21:57.625499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:57.625469 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:58.772243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:58.772207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:59.264109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:59.264075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:59.362366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:59.362326 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:21:59.597577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:21:59.597503 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:00.663198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:00.663167 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb"] Oct 30 03:22:00.663576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:00.663381 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" podUID="d6d4e3e2-f826-4f72-8698-a56ff7211ae5" containerName="maas-api" containerID="cri-o://6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201" gracePeriod=30 Oct 30 03:22:00.725728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:00.725695 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:00.996806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:00.996784 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:22:01.068230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.068197 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls\") pod \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " Oct 30 03:22:01.068408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.068251 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-kube-api-access-vhwmk\" (UniqueName: \"kubernetes.io/projected/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-kube-api-access-vhwmk\") pod \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " Oct 30 03:22:01.068408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.068366 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-openshift-service-ca\") pod \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " Oct 30 03:22:01.068535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.068508 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-metrics-certs\") pod \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\" (UID: \"d6d4e3e2-f826-4f72-8698-a56ff7211ae5\") " Oct 30 03:22:01.068936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.068903 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-openshift-service-ca" pod "d6d4e3e2-f826-4f72-8698-a56ff7211ae5" (UID: "d6d4e3e2-f826-4f72-8698-a56ff7211ae5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:01.070284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.070253 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:01.071419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.071388 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-metrics-certs" pod "d6d4e3e2-f826-4f72-8698-a56ff7211ae5" (UID: "d6d4e3e2-f826-4f72-8698-a56ff7211ae5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:22:01.071531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.071426 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-kube-api-access-vhwmk" pod "d6d4e3e2-f826-4f72-8698-a56ff7211ae5" (UID: "d6d4e3e2-f826-4f72-8698-a56ff7211ae5"). InnerVolumeSpecName "kube-api-access-vhwmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:01.071624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.071410 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls" pod "d6d4e3e2-f826-4f72-8698-a56ff7211ae5" (UID: "d6d4e3e2-f826-4f72-8698-a56ff7211ae5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:22:01.204777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.188609 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:01.204777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.188668 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:01.204777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.188690 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwmk\" (UniqueName: \"kubernetes.io/projected/d6d4e3e2-f826-4f72-8698-a56ff7211ae5-kube-api-access-vhwmk\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:01.285032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.284991 4442 generic.go:356] "Generic (PLEG): container finished" podID="d6d4e3e2-f826-4f72-8698-a56ff7211ae5" containerID="6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201" exitCode=0 Oct 30 03:22:01.285212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.285055 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" Oct 30 03:22:01.285212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.285075 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" event={"ID":"d6d4e3e2-f826-4f72-8698-a56ff7211ae5","Type":"ContainerDied","Data":"6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201"} Oct 30 03:22:01.285212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.285128 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb" event={"ID":"d6d4e3e2-f826-4f72-8698-a56ff7211ae5","Type":"ContainerDied","Data":"a60394243a378a5c1aad0586aec9d5f144b5103545cbbefe9c2d11392f218577"} Oct 30 03:22:01.285212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.285149 4442 scope.go:122] "RemoveContainer" containerID="6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201" Oct 30 03:22:01.297264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.297230 4442 scope.go:122] "RemoveContainer" containerID="6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201" Oct 30 03:22:01.297636 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:01.297613 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201\": container with ID starting with 6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201 not found: ID does not exist" containerID="6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201" Oct 30 03:22:01.297729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.297648 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201"} err="failed to get container status \"6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201\": rpc error: code = NotFound desc = could not find container \"6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201\": container with ID starting with 6e95c934eb8e4e592861cb9ec3a2344ee236487a383002a95bd2cb6009f30201 not found: ID does not exist" Oct 30 03:22:01.331625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.331599 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb"] Oct 30 03:22:01.342047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.342001 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3b93b5-84b7bfbd8d-nsbbb"] Oct 30 03:22:01.389440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:01.389414 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:02.106693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:02.106658 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d4e3e2-f826-4f72-8698-a56ff7211ae5" path="/var/lib/kubelet/pods/d6d4e3e2-f826-4f72-8698-a56ff7211ae5/volumes" Oct 30 03:22:02.268346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:02.268320 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:22:03.175505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:03.175475 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:03.495230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:03.495151 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:03.596633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:03.596605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:05.180805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.180768 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:05.251691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.251659 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m"] Oct 30 03:22:05.252140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.252127 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d6d4e3e2-f826-4f72-8698-a56ff7211ae5" containerName="maas-api" Oct 30 03:22:05.252190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.252142 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d4e3e2-f826-4f72-8698-a56ff7211ae5" containerName="maas-api" Oct 30 03:22:05.252229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.252224 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d4e3e2-f826-4f72-8698-a56ff7211ae5" containerName="maas-api" Oct 30 03:22:05.256914 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.256886 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.276501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.276472 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m"] Oct 30 03:22:05.284147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.284124 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-04fdf3\"" Oct 30 03:22:05.284295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.284274 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-04fdf3-dockercfg-vbk5m\"" Oct 30 03:22:05.315466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.315434 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:05.332807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.332783 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqj2j\" (UniqueName: \"kubernetes.io/projected/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-kube-api-access-vqj2j\") pod \"payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.332920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.332862 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.443574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.434565 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqj2j\" (UniqueName: \"kubernetes.io/projected/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-kube-api-access-vqj2j\") pod \"payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.455781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.454938 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.460024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.456035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.482901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.482872 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqj2j\" (UniqueName: \"kubernetes.io/projected/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-kube-api-access-vqj2j\") pod \"payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.570336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.570300 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:05.872805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:05.872747 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m"] Oct 30 03:22:06.320218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:06.320179 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" event={"ID":"06690641-17cc-4bf3-9c6f-e5f48b6fe7df","Type":"ContainerStarted","Data":"2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8"} Oct 30 03:22:06.320691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:06.320243 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" event={"ID":"06690641-17cc-4bf3-9c6f-e5f48b6fe7df","Type":"ContainerStarted","Data":"cb1b76a7b88cd790f9effd1072b3692fe815a95124d3b13949d3b386fe6957c3"} Oct 30 03:22:06.320691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:06.320280 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:06.358612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:06.358554 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" podStartSLOduration=1.358536365 podStartE2EDuration="1.358536365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:06.357484803 +0000 UTC m=+1150.359837999" watchObservedRunningTime="2026-10-30 03:22:06.358536365 +0000 UTC m=+1150.360889562" Oct 30 03:22:06.962241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:06.962207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:07.074598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:07.074564 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:07.680995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:07.680962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:08.050296 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:08.050261 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:08.908970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:08.908897 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:09.032568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:09.032536 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:09.955776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:09.955718 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:10.071885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:10.071853 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:10.616500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:10.616458 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:12.507440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:12.507398 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:14.207327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:14.207293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:15.662061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:15.662028 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n"] Oct 30 03:22:15.662516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:15.662286 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" podUID="38ba7153-5eb1-4254-8b4b-81daaf6ae865" containerName="maas-api" containerID="cri-o://1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107" gracePeriod=30 Oct 30 03:22:15.733075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:15.733042 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:15.774994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:15.774691 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" podUID="38ba7153-5eb1-4254-8b4b-81daaf6ae865" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.72:8443/health\": dial tcp 10.134.0.72:8443: connect: connection refused" Oct 30 03:22:15.833394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:15.833356 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:15.930484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:15.930420 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:22:16.003943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.003908 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:16.102584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.102555 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls\") pod \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " Oct 30 03:22:16.102805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.102597 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ba7153-5eb1-4254-8b4b-81daaf6ae865-kube-api-access-d29lt\" (UniqueName: \"kubernetes.io/projected/38ba7153-5eb1-4254-8b4b-81daaf6ae865-kube-api-access-d29lt\") pod \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " Oct 30 03:22:16.102805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.102694 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-metrics-certs\") pod \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " Oct 30 03:22:16.102805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.102785 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ba7153-5eb1-4254-8b4b-81daaf6ae865-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ba7153-5eb1-4254-8b4b-81daaf6ae865-openshift-service-ca\") pod \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\" (UID: \"38ba7153-5eb1-4254-8b4b-81daaf6ae865\") " Oct 30 03:22:16.103448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.103414 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ba7153-5eb1-4254-8b4b-81daaf6ae865-openshift-service-ca" pod "38ba7153-5eb1-4254-8b4b-81daaf6ae865" (UID: "38ba7153-5eb1-4254-8b4b-81daaf6ae865"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:16.103700 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.103680 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38ba7153-5eb1-4254-8b4b-81daaf6ae865-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:16.105360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.105326 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ba7153-5eb1-4254-8b4b-81daaf6ae865-kube-api-access-d29lt" pod "38ba7153-5eb1-4254-8b4b-81daaf6ae865" (UID: "38ba7153-5eb1-4254-8b4b-81daaf6ae865"). InnerVolumeSpecName "kube-api-access-d29lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:16.105694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.105607 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-metrics-certs" pod "38ba7153-5eb1-4254-8b4b-81daaf6ae865" (UID: "38ba7153-5eb1-4254-8b4b-81daaf6ae865"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:22:16.105818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.105718 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls" pod "38ba7153-5eb1-4254-8b4b-81daaf6ae865" (UID: "38ba7153-5eb1-4254-8b4b-81daaf6ae865"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:22:16.112063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.112025 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:16.227898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.225240 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:16.227898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.226359 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:16.227898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.226386 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29lt\" (UniqueName: \"kubernetes.io/projected/38ba7153-5eb1-4254-8b4b-81daaf6ae865-kube-api-access-d29lt\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:16.227898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.226401 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38ba7153-5eb1-4254-8b4b-81daaf6ae865-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:16.327359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.327328 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:16.375514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.375471 4442 generic.go:356] "Generic (PLEG): container finished" podID="38ba7153-5eb1-4254-8b4b-81daaf6ae865" containerID="1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107" exitCode=0 Oct 30 03:22:16.375660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.375535 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" event={"ID":"38ba7153-5eb1-4254-8b4b-81daaf6ae865","Type":"ContainerDied","Data":"1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107"} Oct 30 03:22:16.375660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.375580 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" Oct 30 03:22:16.375660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.375594 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n" event={"ID":"38ba7153-5eb1-4254-8b4b-81daaf6ae865","Type":"ContainerDied","Data":"b960c514f76f217ee8179ff99ab8f34e13c46460fd7d9a09c12fbdfceaa406ab"} Oct 30 03:22:16.375660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.375629 4442 scope.go:122] "RemoveContainer" containerID="1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107" Oct 30 03:22:16.386057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.386036 4442 scope.go:122] "RemoveContainer" containerID="1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107" Oct 30 03:22:16.386360 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:16.386337 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107\": container with ID starting with 1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107 not found: ID does not exist" containerID="1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107" Oct 30 03:22:16.386449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.386366 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107"} err="failed to get container status \"1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107\": rpc error: code = NotFound desc = could not find container \"1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107\": container with ID starting with 1a1944e28c79524ddc68a63372e42394d453636032ecfb2c4bebfc78bc133107 not found: ID does not exist" Oct 30 03:22:16.424577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.424538 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n"] Oct 30 03:22:16.433654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.433624 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b132360f-56ccc6c4d6-dqg4n"] Oct 30 03:22:16.461426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.461397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:16.592142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:16.592102 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:17.329526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:17.329496 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:17.541538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:17.541507 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss"] Oct 30 03:22:17.677181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:17.677105 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:17.877726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:17.877700 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:17.977868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:17.977782 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:18.160783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.151242 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ba7153-5eb1-4254-8b4b-81daaf6ae865" path="/var/lib/kubelet/pods/38ba7153-5eb1-4254-8b4b-81daaf6ae865/volumes" Oct 30 03:22:18.386931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.386900 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt"] Oct 30 03:22:18.387647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.387624 4442 cpu_manager.go:414] "containerMap: removing container" podUID="38ba7153-5eb1-4254-8b4b-81daaf6ae865" containerName="maas-api" Oct 30 03:22:18.387647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.387647 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ba7153-5eb1-4254-8b4b-81daaf6ae865" containerName="maas-api" Oct 30 03:22:18.387877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.387795 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ba7153-5eb1-4254-8b4b-81daaf6ae865" containerName="maas-api" Oct 30 03:22:18.392785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.392765 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.417913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.417887 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-04fdf3-serving-cert\"" Oct 30 03:22:18.420234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.420200 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt"] Oct 30 03:22:18.465725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.465699 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.465881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.465829 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-metrics-certs\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.465926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.465882 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfz9k\" (UniqueName: \"kubernetes.io/projected/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-kube-api-access-hfz9k\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.465996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.465978 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.474687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.474663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:18.565982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.565956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:18.567161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.567140 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.567223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.567205 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.567291 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:18.567235 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-serving-cert: secret "maas-api-e2e-ipp-a-04fdf3-serving-cert" not found Oct 30 03:22:18.567291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.567267 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-metrics-certs\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.567291 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:18.567284 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls podName:d3af3ca2-1311-4de4-a94f-2fd1b5a5e479 nodeName:}" failed. No retries permitted until 2026-10-30 03:22:19.067267123 +0000 UTC m=+1163.069620315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls") pod "maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" (UID: "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479") : secret "maas-api-e2e-ipp-a-04fdf3-serving-cert" not found Oct 30 03:22:18.567443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.567332 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfz9k\" (UniqueName: \"kubernetes.io/projected/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-kube-api-access-hfz9k\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.567922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.567898 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.569980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.569957 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-metrics-certs\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.615777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.614862 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfz9k\" (UniqueName: \"kubernetes.io/projected/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-kube-api-access-hfz9k\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:18.946727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:18.946699 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:19.071431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:19.071399 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:19.074041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:19.074019 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls\") pod \"maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:19.306543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:19.306513 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:19.651728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:19.651695 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt"] Oct 30 03:22:19.652540 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:22:19.652513 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3af3ca2_1311_4de4_a94f_2fd1b5a5e479.slice/crio-cae9dd8a67a17cfc7a6dda6ab3d7ab4104df0963919614e591f75038c7bc29a0 WatchSource:0}: Error finding container cae9dd8a67a17cfc7a6dda6ab3d7ab4104df0963919614e591f75038c7bc29a0: Status 404 returned error can't find the container with id cae9dd8a67a17cfc7a6dda6ab3d7ab4104df0963919614e591f75038c7bc29a0 Oct 30 03:22:20.325513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:20.325431 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:20.419567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:20.419531 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" event={"ID":"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479","Type":"ContainerStarted","Data":"ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab"} Oct 30 03:22:20.419806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:20.419597 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" event={"ID":"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479","Type":"ContainerStarted","Data":"cae9dd8a67a17cfc7a6dda6ab3d7ab4104df0963919614e591f75038c7bc29a0"} Oct 30 03:22:20.419806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:20.419710 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:20.460098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:20.460036 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" podStartSLOduration=2.087586836 podStartE2EDuration="2.460017877s" totalImagesPullingTime="372.431041ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:20.458687532 +0000 UTC m=+1164.461040731" watchObservedRunningTime="2026-10-30 03:22:20.460017877 +0000 UTC m=+1164.462371074" Oct 30 03:22:20.795207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:20.795175 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:21.091910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:21.091839 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:22.056320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.056293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:22.377738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.377652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:22.478232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.478201 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:22.542213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.542182 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" podUID="9778eefd-ec5d-4ed8-898c-6dd2265544d7" containerName="payload-pre-processing" containerID="cri-o://9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4" gracePeriod=25 Oct 30 03:22:22.766090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.766056 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:22.817984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.817955 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:22:22.844178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.844132 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9778eefd-ec5d-4ed8-898c-6dd2265544d7-kube-api-access-jfgn7\" (UniqueName: \"kubernetes.io/projected/9778eefd-ec5d-4ed8-898c-6dd2265544d7-kube-api-access-jfgn7\") pod \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " Oct 30 03:22:22.844334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.844233 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9778eefd-ec5d-4ed8-898c-6dd2265544d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9778eefd-ec5d-4ed8-898c-6dd2265544d7-plugins-config-volume\") pod \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\" (UID: \"9778eefd-ec5d-4ed8-898c-6dd2265544d7\") " Oct 30 03:22:22.844673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.844641 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9778eefd-ec5d-4ed8-898c-6dd2265544d7-plugins-config-volume" pod "9778eefd-ec5d-4ed8-898c-6dd2265544d7" (UID: "9778eefd-ec5d-4ed8-898c-6dd2265544d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:22.846554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.846523 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9778eefd-ec5d-4ed8-898c-6dd2265544d7-kube-api-access-jfgn7" pod "9778eefd-ec5d-4ed8-898c-6dd2265544d7" (UID: "9778eefd-ec5d-4ed8-898c-6dd2265544d7"). InnerVolumeSpecName "kube-api-access-jfgn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:22.945802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.945703 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfgn7\" (UniqueName: \"kubernetes.io/projected/9778eefd-ec5d-4ed8-898c-6dd2265544d7-kube-api-access-jfgn7\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:22.945802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:22.945729 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9778eefd-ec5d-4ed8-898c-6dd2265544d7-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:23.062993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.062960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:23.439162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.439127 4442 generic.go:356] "Generic (PLEG): container finished" podID="9778eefd-ec5d-4ed8-898c-6dd2265544d7" containerID="9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4" exitCode=0 Oct 30 03:22:23.439333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.439171 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" Oct 30 03:22:23.439333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.439199 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" event={"ID":"9778eefd-ec5d-4ed8-898c-6dd2265544d7","Type":"ContainerDied","Data":"9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4"} Oct 30 03:22:23.439333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.439250 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss" event={"ID":"9778eefd-ec5d-4ed8-898c-6dd2265544d7","Type":"ContainerDied","Data":"54efc9253c9226ee34af6a510d638502b31d5518c1994e5c3fd5260983b424d1"} Oct 30 03:22:23.439333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.439278 4442 scope.go:122] "RemoveContainer" containerID="9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4" Oct 30 03:22:23.454391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.454353 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:23.460274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.460251 4442 scope.go:122] "RemoveContainer" containerID="9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4" Oct 30 03:22:23.460595 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:23.460565 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4\": container with ID starting with 9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4 not found: ID does not exist" containerID="9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4" Oct 30 03:22:23.460708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.460590 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4"} err="failed to get container status \"9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4\": rpc error: code = NotFound desc = could not find container \"9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4\": container with ID starting with 9a9e4a489b0656d852d675b64d6d957636cb8b0b614353264e5206aab1156eb4 not found: ID does not exist" Oct 30 03:22:23.497310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.497281 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss"] Oct 30 03:22:23.507886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.507862 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-d6c536-574b6688d7-l45ss"] Oct 30 03:22:23.625987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.625180 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:22:23.907540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:23.907511 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:24.106256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:24.106227 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9778eefd-ec5d-4ed8-898c-6dd2265544d7" path="/var/lib/kubelet/pods/9778eefd-ec5d-4ed8-898c-6dd2265544d7/volumes" Oct 30 03:22:24.861193 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:24.861148 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:22:25.428325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.428284 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw"] Oct 30 03:22:25.428835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.428546 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" podUID="048fe785-48cf-4594-ab8c-a2b65cc87eeb" containerName="maas-api" containerID="cri-o://5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432" gracePeriod=30 Oct 30 03:22:25.474876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.474847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:25.761720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.761696 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:22:25.779065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.779039 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-maas-api-tls\") pod \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " Oct 30 03:22:25.779273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.779095 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048fe785-48cf-4594-ab8c-a2b65cc87eeb-kube-api-access-sspk9\" (UniqueName: \"kubernetes.io/projected/048fe785-48cf-4594-ab8c-a2b65cc87eeb-kube-api-access-sspk9\") pod \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " Oct 30 03:22:25.779273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.779131 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048fe785-48cf-4594-ab8c-a2b65cc87eeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048fe785-48cf-4594-ab8c-a2b65cc87eeb-openshift-service-ca\") pod \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " Oct 30 03:22:25.779273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.779206 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-metrics-certs\") pod \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\" (UID: \"048fe785-48cf-4594-ab8c-a2b65cc87eeb\") " Oct 30 03:22:25.779538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.779504 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048fe785-48cf-4594-ab8c-a2b65cc87eeb-openshift-service-ca" pod "048fe785-48cf-4594-ab8c-a2b65cc87eeb" (UID: "048fe785-48cf-4594-ab8c-a2b65cc87eeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:25.781340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.781315 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048fe785-48cf-4594-ab8c-a2b65cc87eeb-kube-api-access-sspk9" pod "048fe785-48cf-4594-ab8c-a2b65cc87eeb" (UID: "048fe785-48cf-4594-ab8c-a2b65cc87eeb"). InnerVolumeSpecName "kube-api-access-sspk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:25.781440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.781315 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-maas-api-tls" pod "048fe785-48cf-4594-ab8c-a2b65cc87eeb" (UID: "048fe785-48cf-4594-ab8c-a2b65cc87eeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:22:25.781440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.781391 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-metrics-certs" pod "048fe785-48cf-4594-ab8c-a2b65cc87eeb" (UID: "048fe785-48cf-4594-ab8c-a2b65cc87eeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:22:25.880654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.880623 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:25.880654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.880650 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/048fe785-48cf-4594-ab8c-a2b65cc87eeb-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:25.880654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.880660 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sspk9\" (UniqueName: \"kubernetes.io/projected/048fe785-48cf-4594-ab8c-a2b65cc87eeb-kube-api-access-sspk9\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:25.880919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:25.880670 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/048fe785-48cf-4594-ab8c-a2b65cc87eeb-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:26.193862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.193823 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:26.430885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.430800 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:22:26.459473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.459444 4442 generic.go:356] "Generic (PLEG): container finished" podID="048fe785-48cf-4594-ab8c-a2b65cc87eeb" containerID="5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432" exitCode=0 Oct 30 03:22:26.459639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.459491 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" Oct 30 03:22:26.459639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.459528 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" event={"ID":"048fe785-48cf-4594-ab8c-a2b65cc87eeb","Type":"ContainerDied","Data":"5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432"} Oct 30 03:22:26.459639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.459579 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw" event={"ID":"048fe785-48cf-4594-ab8c-a2b65cc87eeb","Type":"ContainerDied","Data":"5b44643cd1ea4360c4bb9102d7356fd54475dd90c2578f284a0f3a21e7686ecb"} Oct 30 03:22:26.459639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.459603 4442 scope.go:122] "RemoveContainer" containerID="5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432" Oct 30 03:22:26.468608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.468592 4442 scope.go:122] "RemoveContainer" containerID="5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432" Oct 30 03:22:26.468941 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:26.468907 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432\": container with ID starting with 5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432 not found: ID does not exist" containerID="5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432" Oct 30 03:22:26.469030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.468948 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432"} err="failed to get container status \"5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432\": rpc error: code = NotFound desc = could not find container \"5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432\": container with ID starting with 5df99484f237c8e4ea0b7c3d22109419d173f87dafcd722769b25f9b4d4df432 not found: ID does not exist" Oct 30 03:22:26.506251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.506222 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw"] Oct 30 03:22:26.509964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:26.509934 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d4b0d5-5c5b649cc8-mz9kw"] Oct 30 03:22:27.556557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:27.556521 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:27.668315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:27.668288 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:28.029903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.029867 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74"] Oct 30 03:22:28.030430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.030410 4442 cpu_manager.go:414] "containerMap: removing container" podUID="048fe785-48cf-4594-ab8c-a2b65cc87eeb" containerName="maas-api" Oct 30 03:22:28.030531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.030432 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048fe785-48cf-4594-ab8c-a2b65cc87eeb" containerName="maas-api" Oct 30 03:22:28.030531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.030462 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9778eefd-ec5d-4ed8-898c-6dd2265544d7" containerName="payload-pre-processing" Oct 30 03:22:28.030531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.030486 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9778eefd-ec5d-4ed8-898c-6dd2265544d7" containerName="payload-pre-processing" Oct 30 03:22:28.030685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.030601 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="048fe785-48cf-4594-ab8c-a2b65cc87eeb" containerName="maas-api" Oct 30 03:22:28.030685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.030617 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9778eefd-ec5d-4ed8-898c-6dd2265544d7" containerName="payload-pre-processing" Oct 30 03:22:28.045229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.045209 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.054988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.054966 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74"] Oct 30 03:22:28.062287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.062257 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-d6c536\"" Oct 30 03:22:28.062380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.062362 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-d6c536-dockercfg-pm5vp\"" Oct 30 03:22:28.106009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.105981 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26770af4-ea80-49a5-85c1-a5b1e30883b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.106170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.106147 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9hf\" (UniqueName: \"kubernetes.io/projected/26770af4-ea80-49a5-85c1-a5b1e30883b1-kube-api-access-2k9hf\") pod \"payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.106990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.106973 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048fe785-48cf-4594-ab8c-a2b65cc87eeb" path="/var/lib/kubelet/pods/048fe785-48cf-4594-ab8c-a2b65cc87eeb/volumes" Oct 30 03:22:28.206650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.206620 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26770af4-ea80-49a5-85c1-a5b1e30883b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.206868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.206736 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9hf\" (UniqueName: \"kubernetes.io/projected/26770af4-ea80-49a5-85c1-a5b1e30883b1-kube-api-access-2k9hf\") pod \"payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.207295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.207277 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26770af4-ea80-49a5-85c1-a5b1e30883b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.224722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.224701 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9hf\" (UniqueName: \"kubernetes.io/projected/26770af4-ea80-49a5-85c1-a5b1e30883b1-kube-api-access-2k9hf\") pod \"payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.363917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.363841 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:28.635627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.635550 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:28.663268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:28.663245 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74"] Oct 30 03:22:28.664801 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:22:28.664777 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26770af4_ea80_49a5_85c1_a5b1e30883b1.slice/crio-b72285c7370c1c1b6c806b204bf4e76dd3b0f5c6f1d6c177002523c8288f71be WatchSource:0}: Error finding container b72285c7370c1c1b6c806b204bf4e76dd3b0f5c6f1d6c177002523c8288f71be: Status 404 returned error can't find the container with id b72285c7370c1c1b6c806b204bf4e76dd3b0f5c6f1d6c177002523c8288f71be Oct 30 03:22:29.479512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:29.479468 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" event={"ID":"26770af4-ea80-49a5-85c1-a5b1e30883b1","Type":"ContainerStarted","Data":"b72285c7370c1c1b6c806b204bf4e76dd3b0f5c6f1d6c177002523c8288f71be"} Oct 30 03:22:29.584567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:29.584535 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:29.685447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:29.685382 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:30.158816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.158793 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:30.403875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.403793 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:30.522777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.514826 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:30.533009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.531959 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" event={"ID":"26770af4-ea80-49a5-85c1-a5b1e30883b1","Type":"ContainerStarted","Data":"0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9"} Oct 30 03:22:30.533009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.532933 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:30.576617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.576559 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" podStartSLOduration=1.092169026 podStartE2EDuration="2.57653115s" totalImagesPullingTime="1.484362124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:30.574804897 +0000 UTC m=+1174.577158095" watchObservedRunningTime="2026-10-30 03:22:30.57653115 +0000 UTC m=+1174.578884349" Oct 30 03:22:30.817779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.816400 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj"] Oct 30 03:22:30.817779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.816999 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" containerID="cri-o://7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8" gracePeriod=30 Oct 30 03:22:30.822562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:30.819063 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:31.797168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:31.797127 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:32.251619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:32.251569 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:32.373237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:32.373207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:32.478470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:32.478438 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:34.041337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:34.041306 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:35.929614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:35.929582 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:36.126422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.126394 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:36.131079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.131057 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:22:36.226647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.226616 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83237762-f446-40ca-9500-861aaf8a326e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83237762-f446-40ca-9500-861aaf8a326e-istio-podinfo\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.226842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.226704 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-credential-socket\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.226842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.226745 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-istio-token\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-istio-token\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.226842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.226809 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-envoy\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.227013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.226846 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83237762-f446-40ca-9500-861aaf8a326e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83237762-f446-40ca-9500-861aaf8a326e-istiod-ca-cert\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.227071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227019 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-kube-api-access-2dflp\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-kube-api-access-2dflp\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.227126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227055 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-credential-socket" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:22:36.227126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227089 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-data\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.227233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227132 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-socket\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.227233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227186 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-certs\") pod \"83237762-f446-40ca-9500-861aaf8a326e\" (UID: \"83237762-f446-40ca-9500-861aaf8a326e\") " Oct 30 03:22:36.227351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227233 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83237762-f446-40ca-9500-861aaf8a326e-istiod-ca-cert" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:36.227409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227351 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-data" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:22:36.227853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227626 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-certs" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:22:36.227853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.227667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-socket" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:22:36.228255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.228236 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.228330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.228261 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83237762-f446-40ca-9500-861aaf8a326e-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.228330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.228277 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.228330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.228290 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.228330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.228302 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.229285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.229253 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83237762-f446-40ca-9500-861aaf8a326e-istio-podinfo" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:22:36.229285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.229258 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-envoy" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:22:36.229435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.229331 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-istio-token" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:36.229509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.229488 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-kube-api-access-2dflp" pod "83237762-f446-40ca-9500-861aaf8a326e" (UID: "83237762-f446-40ca-9500-861aaf8a326e"). InnerVolumeSpecName "kube-api-access-2dflp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:36.230416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.230392 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:36.329916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.329878 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dflp\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-kube-api-access-2dflp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.329916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.329909 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83237762-f446-40ca-9500-861aaf8a326e-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.329916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.329922 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83237762-f446-40ca-9500-861aaf8a326e-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.330148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.329933 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83237762-f446-40ca-9500-861aaf8a326e-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:36.583637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.583609 4442 generic.go:356] "Generic (PLEG): container finished" podID="83237762-f446-40ca-9500-861aaf8a326e" containerID="7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8" exitCode=0 Oct 30 03:22:36.583843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.583681 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" Oct 30 03:22:36.583843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.583704 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" event={"ID":"83237762-f446-40ca-9500-861aaf8a326e","Type":"ContainerDied","Data":"7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8"} Oct 30 03:22:36.583843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.583808 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj" event={"ID":"83237762-f446-40ca-9500-861aaf8a326e","Type":"ContainerDied","Data":"701f3c1c3053c0eb5c5349e9c3df38594b5a9bcf54393268ae5c0ac20f313549"} Oct 30 03:22:36.583843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.583833 4442 scope.go:122] "RemoveContainer" containerID="7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8" Oct 30 03:22:36.593673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.593660 4442 scope.go:122] "RemoveContainer" containerID="49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b" Oct 30 03:22:36.601961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.601941 4442 scope.go:122] "RemoveContainer" containerID="7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8" Oct 30 03:22:36.602213 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:36.602195 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8\": container with ID starting with 7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8 not found: ID does not exist" containerID="7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8" Oct 30 03:22:36.602264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.602221 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8"} err="failed to get container status \"7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8\": rpc error: code = NotFound desc = could not find container \"7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8\": container with ID starting with 7dc85ddd040f82234b49e98674d9bb3825cfae228ab56241c2b2e48604af11e8 not found: ID does not exist" Oct 30 03:22:36.602264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.602239 4442 scope.go:122] "RemoveContainer" containerID="49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b" Oct 30 03:22:36.602474 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:36.602454 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b\": container with ID starting with 49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b not found: ID does not exist" containerID="49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b" Oct 30 03:22:36.602516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.602482 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b"} err="failed to get container status \"49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b\": rpc error: code = NotFound desc = could not find container \"49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b\": container with ID starting with 49076105eedee3a4c8a6faa415e1684ad87b0bad0c27b7b6bd4bbea2fbc8548b not found: ID does not exist" Oct 30 03:22:36.634593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.634569 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj"] Oct 30 03:22:36.649133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:36.649108 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b0d5-openshift-default-7dcf44b869-xnwhj"] Oct 30 03:22:37.658481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:37.658449 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:37.757623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:37.757596 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:37.984040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:37.983562 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:38.107409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:38.107377 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83237762-f446-40ca-9500-861aaf8a326e" path="/var/lib/kubelet/pods/83237762-f446-40ca-9500-861aaf8a326e/volumes" Oct 30 03:22:38.950412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:38.950374 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m"] Oct 30 03:22:39.065683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:39.065647 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:39.179484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:39.179445 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:39.288836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:39.288806 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:39.855073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:39.855038 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:41.278706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:41.278671 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:22:41.826187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:41.826157 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:42.249692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:42.249658 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:42.556648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:42.556567 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:22:43.953781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:43.952836 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" podUID="06690641-17cc-4bf3-9c6f-e5f48b6fe7df" containerName="payload-processing" containerID="cri-o://2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8" gracePeriod=25 Oct 30 03:22:44.444923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.420047 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:44.468776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.468483 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:44.577708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.577422 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-plugins-config-volume\") pod \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " Oct 30 03:22:44.578471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.578055 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-kube-api-access-vqj2j\" (UniqueName: \"kubernetes.io/projected/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-kube-api-access-vqj2j\") pod \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\" (UID: \"06690641-17cc-4bf3-9c6f-e5f48b6fe7df\") " Oct 30 03:22:44.578471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.578219 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-plugins-config-volume" pod "06690641-17cc-4bf3-9c6f-e5f48b6fe7df" (UID: "06690641-17cc-4bf3-9c6f-e5f48b6fe7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:44.578801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.578781 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:44.581229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.581200 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-kube-api-access-vqj2j" pod "06690641-17cc-4bf3-9c6f-e5f48b6fe7df" (UID: "06690641-17cc-4bf3-9c6f-e5f48b6fe7df"). InnerVolumeSpecName "kube-api-access-vqj2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:44.598539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.598517 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:44.660033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.660002 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:22:44.660322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.660275 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:22:44.675918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.675876 4442 generic.go:356] "Generic (PLEG): container finished" podID="06690641-17cc-4bf3-9c6f-e5f48b6fe7df" containerID="2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8" exitCode=0 Oct 30 03:22:44.676031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.675918 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" event={"ID":"06690641-17cc-4bf3-9c6f-e5f48b6fe7df","Type":"ContainerDied","Data":"2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8"} Oct 30 03:22:44.676031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.675969 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" Oct 30 03:22:44.676031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.675983 4442 scope.go:122] "RemoveContainer" containerID="2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8" Oct 30 03:22:44.676155 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.675970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m" event={"ID":"06690641-17cc-4bf3-9c6f-e5f48b6fe7df","Type":"ContainerDied","Data":"cb1b76a7b88cd790f9effd1072b3692fe815a95124d3b13949d3b386fe6957c3"} Oct 30 03:22:44.679708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.679686 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqj2j\" (UniqueName: \"kubernetes.io/projected/06690641-17cc-4bf3-9c6f-e5f48b6fe7df-kube-api-access-vqj2j\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:22:44.716978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.713143 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:44.726489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.725558 4442 scope.go:122] "RemoveContainer" containerID="2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8" Oct 30 03:22:44.726718 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:44.726659 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8\": container with ID starting with 2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8 not found: ID does not exist" containerID="2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8" Oct 30 03:22:44.726718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.726686 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8"} err="failed to get container status \"2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8\": rpc error: code = NotFound desc = could not find container \"2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8\": container with ID starting with 2273b23fff7464db3c27b91cd2c1632d74804bc899dfc7825b681395d31b94d8 not found: ID does not exist" Oct 30 03:22:44.760886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.760856 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m"] Oct 30 03:22:44.766078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:44.765662 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-04fdf3-7f9f845fd6-zpr7m"] Oct 30 03:22:45.011509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.011477 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5"] Oct 30 03:22:45.012224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.012204 4442 cpu_manager.go:414] "containerMap: removing container" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" Oct 30 03:22:45.012269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.012228 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" Oct 30 03:22:45.012269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.012262 4442 cpu_manager.go:414] "containerMap: removing container" podUID="06690641-17cc-4bf3-9c6f-e5f48b6fe7df" containerName="payload-processing" Oct 30 03:22:45.012333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.012270 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06690641-17cc-4bf3-9c6f-e5f48b6fe7df" containerName="payload-processing" Oct 30 03:22:45.012397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.012386 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" Oct 30 03:22:45.012438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.012406 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" Oct 30 03:22:45.012438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.012418 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="06690641-17cc-4bf3-9c6f-e5f48b6fe7df" containerName="payload-processing" Oct 30 03:22:45.017527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.017503 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.044764 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.044714 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a1e5d5-openshift-default-dockercfg-n9ffz\"" Oct 30 03:22:45.049139 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.049113 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5"] Oct 30 03:22:45.085807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.085763 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-podinfo\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.085955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.085849 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-token\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.085955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.085882 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istiod-ca-cert\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.085955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.085907 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-credential-socket\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.086109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.085986 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-socket\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.086109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.086063 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz29w\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-kube-api-access-dz29w\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.086109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.086102 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-certs\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.086230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.086159 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-data\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.086230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.086198 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-envoy\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.157490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.157466 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:45.186917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.186895 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-podinfo\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.186935 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-token\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.186959 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istiod-ca-cert\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.186974 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-credential-socket\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187002 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-socket\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187365 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-socket\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187438 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-credential-socket\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187454 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz29w\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-kube-api-access-dz29w\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187495 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-certs\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187553 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-data\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187582 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-envoy\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187830 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istiod-ca-cert\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.187985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.187962 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-certs\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.188069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.188006 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-data\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.189866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.189835 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-podinfo\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.190180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.190155 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-envoy\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.209391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.209335 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-token\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.221632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.221546 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz29w\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-kube-api-access-dz29w\") pod \"e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.271972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.271716 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:45.351643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.351608 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:45.665983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.665952 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5"] Oct 30 03:22:45.667070 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:22:45.667044 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904c0e21_cf94_46fd_8cfc_e4edaf0105b3.slice/crio-b43e9829250036bd277903777f69f9499f2ce45c0d7fee0111ba9f66d0ba95b9 WatchSource:0}: Error finding container b43e9829250036bd277903777f69f9499f2ce45c0d7fee0111ba9f66d0ba95b9: Status 404 returned error can't find the container with id b43e9829250036bd277903777f69f9499f2ce45c0d7fee0111ba9f66d0ba95b9 Oct 30 03:22:45.669491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.669462 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:22:45.669575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.669524 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:22:45.669575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.669554 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:22:45.684564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:45.684536 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" event={"ID":"904c0e21-cf94-46fd-8cfc-e4edaf0105b3","Type":"ContainerStarted","Data":"b43e9829250036bd277903777f69f9499f2ce45c0d7fee0111ba9f66d0ba95b9"} Oct 30 03:22:46.106850 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:46.106818 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06690641-17cc-4bf3-9c6f-e5f48b6fe7df" path="/var/lib/kubelet/pods/06690641-17cc-4bf3-9c6f-e5f48b6fe7df/volumes" Oct 30 03:22:46.413267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:46.413189 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:46.693586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:46.693513 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" event={"ID":"904c0e21-cf94-46fd-8cfc-e4edaf0105b3","Type":"ContainerStarted","Data":"2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58"} Oct 30 03:22:46.756893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:46.756845 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podStartSLOduration=2.756830109 podStartE2EDuration="2.756830109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:46.744346723 +0000 UTC m=+1190.746699916" watchObservedRunningTime="2026-10-30 03:22:46.756830109 +0000 UTC m=+1190.759183306" Oct 30 03:22:47.352654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:47.352625 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:47.358196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:47.358170 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:47.699584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:47.699510 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:47.700565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:47.700542 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:22:48.440118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:48.440088 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:49.006459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:49.006428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:49.116967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:49.116930 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:49.233955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:49.233928 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:50.487481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.487405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:50.814450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.814421 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:50.910938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.910904 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:50.944412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.944386 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm"] Oct 30 03:22:50.944942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.944924 4442 cpu_manager.go:414] "containerMap: removing container" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" Oct 30 03:22:50.945030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.944944 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83237762-f446-40ca-9500-861aaf8a326e" containerName="istio-proxy" Oct 30 03:22:50.948579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.948560 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:50.953810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.953789 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b0298-6579-4261-af0e-29ef71b97b91-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:50.953931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.953823 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rnnp\" (UniqueName: \"kubernetes.io/projected/f07b0298-6579-4261-af0e-29ef71b97b91-kube-api-access-4rnnp\") pod \"payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:50.968267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.968238 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm"] Oct 30 03:22:50.975545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.975516 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a1e5d5\"" Oct 30 03:22:50.975904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:50.975883 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a1e5d5-dockercfg-p27w9\"" Oct 30 03:22:51.067860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.067744 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b0298-6579-4261-af0e-29ef71b97b91-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:51.069485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.069458 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnnp\" (UniqueName: \"kubernetes.io/projected/f07b0298-6579-4261-af0e-29ef71b97b91-kube-api-access-4rnnp\") pod \"payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:51.098226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.095854 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b0298-6579-4261-af0e-29ef71b97b91-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:51.117644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.117613 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnnp\" (UniqueName: \"kubernetes.io/projected/f07b0298-6579-4261-af0e-29ef71b97b91-kube-api-access-4rnnp\") pod \"payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:51.262290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.262263 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:51.570281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.570254 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm"] Oct 30 03:22:51.572167 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:22:51.572110 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07b0298_6579_4261_af0e_29ef71b97b91.slice/crio-3ee06cac9ad9f58630d1cf80cb421291af39de328c0539240c4b7418389a9056 WatchSource:0}: Error finding container 3ee06cac9ad9f58630d1cf80cb421291af39de328c0539240c4b7418389a9056: Status 404 returned error can't find the container with id 3ee06cac9ad9f58630d1cf80cb421291af39de328c0539240c4b7418389a9056 Oct 30 03:22:51.722926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.722892 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" event={"ID":"f07b0298-6579-4261-af0e-29ef71b97b91","Type":"ContainerStarted","Data":"e5528332105cebddca414e34280e4680fadb4f865ddb13d1538a96773498bbe4"} Oct 30 03:22:51.723060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.722945 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" event={"ID":"f07b0298-6579-4261-af0e-29ef71b97b91","Type":"ContainerStarted","Data":"3ee06cac9ad9f58630d1cf80cb421291af39de328c0539240c4b7418389a9056"} Oct 30 03:22:51.723133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.723092 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:22:51.765538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.765495 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" podStartSLOduration=1.7654828230000001 podStartE2EDuration="1.765482823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:51.765200815 +0000 UTC m=+1195.767554039" watchObservedRunningTime="2026-10-30 03:22:51.765482823 +0000 UTC m=+1195.767836020" Oct 30 03:22:51.896140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:51.896054 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:52.057899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:52.051239 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:52.906680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:52.906643 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4"] Oct 30 03:22:52.910653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:52.910634 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:52.924301 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:52.924283 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-499bca-serving-cert\"" Oct 30 03:22:52.968798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:52.968746 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4"] Oct 30 03:22:53.005996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.005962 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.006139 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.006023 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaa8e42-da33-4f8d-a727-3e30e729f934-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.006178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.006150 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-metrics-certs\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.006233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.006191 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65vn\" (UniqueName: \"kubernetes.io/projected/abaa8e42-da33-4f8d-a727-3e30e729f934-kube-api-access-c65vn\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.107827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.107792 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.108029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.107868 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaa8e42-da33-4f8d-a727-3e30e729f934-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.108029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.107921 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-metrics-certs\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.108029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.107944 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65vn\" (UniqueName: \"kubernetes.io/projected/abaa8e42-da33-4f8d-a727-3e30e729f934-kube-api-access-c65vn\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.108565 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:53.108483 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-serving-cert: secret "maas-api-e2e-ipp-a-499bca-serving-cert" not found Oct 30 03:22:53.108696 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:53.108573 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls podName:abaa8e42-da33-4f8d-a727-3e30e729f934 nodeName:}" failed. No retries permitted until 2026-10-30 03:22:53.608554063 +0000 UTC m=+1197.610907244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls") pod "maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" (UID: "abaa8e42-da33-4f8d-a727-3e30e729f934") : secret "maas-api-e2e-ipp-a-499bca-serving-cert" not found Oct 30 03:22:53.108696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.108673 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:53.125779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.122084 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaa8e42-da33-4f8d-a727-3e30e729f934-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.135780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.132369 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-metrics-certs\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.150184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.149434 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65vn\" (UniqueName: \"kubernetes.io/projected/abaa8e42-da33-4f8d-a727-3e30e729f934-kube-api-access-c65vn\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.240863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.240782 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:53.346101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.346074 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:53.461137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.461104 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:53.560952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.560926 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:53.613061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.613029 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.615553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.615528 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls\") pod \"maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:53.823218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:53.823128 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:54.151205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:54.151175 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4"] Oct 30 03:22:54.152327 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:22:54.152302 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabaa8e42_da33_4f8d_a727_3e30e729f934.slice/crio-16f0bcf1559bcdcaf46c7924044bd141428a49283a661b8c20d2a53e674505a1 WatchSource:0}: Error finding container 16f0bcf1559bcdcaf46c7924044bd141428a49283a661b8c20d2a53e674505a1: Status 404 returned error can't find the container with id 16f0bcf1559bcdcaf46c7924044bd141428a49283a661b8c20d2a53e674505a1 Oct 30 03:22:54.758663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:54.758579 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" event={"ID":"abaa8e42-da33-4f8d-a727-3e30e729f934","Type":"ContainerStarted","Data":"6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841"} Oct 30 03:22:54.758663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:54.758641 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" event={"ID":"abaa8e42-da33-4f8d-a727-3e30e729f934","Type":"ContainerStarted","Data":"16f0bcf1559bcdcaf46c7924044bd141428a49283a661b8c20d2a53e674505a1"} Oct 30 03:22:54.758844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:54.758666 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:22:54.808817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:54.808734 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" podStartSLOduration=2.457298645 podStartE2EDuration="2.80871486s" totalImagesPullingTime="351.416215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:22:54.807446736 +0000 UTC m=+1198.809799924" watchObservedRunningTime="2026-10-30 03:22:54.80871486 +0000 UTC m=+1198.811068058" Oct 30 03:22:54.829723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:54.829699 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:54.862018 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:22:54.861989 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:22:56.231247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:56.231222 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:22:56.232911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:56.232888 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:22:56.415220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:56.415192 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:56.533070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:56.533039 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:56.917597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:56.917508 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:57.061518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:57.058624 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:22:58.324291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:22:58.324253 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:00.279249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:00.279211 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:00.367499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:00.367465 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:00.770099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:00.770073 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:23:01.916797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:01.916748 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:02.039407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.039377 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:02.046576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.046550 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs"] Oct 30 03:23:02.053039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.053022 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.071525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.071504 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6c106c\"" Oct 30 03:23:02.082407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.082389 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6c106c-dockercfg-gf2hs\"" Oct 30 03:23:02.095977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.095953 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs"] Oct 30 03:23:02.133915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.133886 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f29b4\" (UniqueName: \"kubernetes.io/projected/8d03adf8-ce28-414e-8126-4a495919ca0b-kube-api-access-f29b4\") pod \"payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.134109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.133931 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d03adf8-ce28-414e-8126-4a495919ca0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.255443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.255407 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f29b4\" (UniqueName: \"kubernetes.io/projected/8d03adf8-ce28-414e-8126-4a495919ca0b-kube-api-access-f29b4\") pod \"payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.256154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.256126 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d03adf8-ce28-414e-8126-4a495919ca0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.257323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.257149 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d03adf8-ce28-414e-8126-4a495919ca0b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.278776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.278732 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f29b4\" (UniqueName: \"kubernetes.io/projected/8d03adf8-ce28-414e-8126-4a495919ca0b-kube-api-access-f29b4\") pod \"payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.365746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.365712 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.664929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.664899 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs"] Oct 30 03:23:02.666491 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:23:02.666464 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d03adf8_ce28_414e_8126_4a495919ca0b.slice/crio-d0ca419387c82e569430cc9ad0da611141ab897838f22903a019c9afda7a5bb4 WatchSource:0}: Error finding container d0ca419387c82e569430cc9ad0da611141ab897838f22903a019c9afda7a5bb4: Status 404 returned error can't find the container with id d0ca419387c82e569430cc9ad0da611141ab897838f22903a019c9afda7a5bb4 Oct 30 03:23:02.733087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.733067 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:23:02.805483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.805455 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" event={"ID":"8d03adf8-ce28-414e-8126-4a495919ca0b","Type":"ContainerStarted","Data":"02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5"} Oct 30 03:23:02.805637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.805507 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" event={"ID":"8d03adf8-ce28-414e-8126-4a495919ca0b","Type":"ContainerStarted","Data":"d0ca419387c82e569430cc9ad0da611141ab897838f22903a019c9afda7a5bb4"} Oct 30 03:23:02.805637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.805606 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:02.843342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.843296 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" podStartSLOduration=0.843281354 podStartE2EDuration="843.281354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:02.841626367 +0000 UTC m=+1206.843979576" watchObservedRunningTime="2026-10-30 03:23:02.843281354 +0000 UTC m=+1206.845634551" Oct 30 03:23:02.920627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:02.920554 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:04.822083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:04.822051 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:05.573383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:05.573341 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:05.965909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:05.965814 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:06.350430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.350393 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8"] Oct 30 03:23:06.355541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.355521 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.369616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.369596 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a1e5d5-serving-cert\"" Oct 30 03:23:06.385026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.385000 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8"] Oct 30 03:23:06.440972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.440940 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:06.505477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.505446 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-metrics-certs\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.505661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.505570 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s55v\" (UniqueName: \"kubernetes.io/projected/8c6d6950-7765-45f9-b233-c7351302d7a9-kube-api-access-8s55v\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.505661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.505638 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.505778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.505721 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d6950-7765-45f9-b233-c7351302d7a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.542541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.542512 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:06.606434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.606355 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s55v\" (UniqueName: \"kubernetes.io/projected/8c6d6950-7765-45f9-b233-c7351302d7a9-kube-api-access-8s55v\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.606434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.606402 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.606661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.606436 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d6950-7765-45f9-b233-c7351302d7a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.606661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.606511 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-metrics-certs\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.606661 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:23:06.606560 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-serving-cert: secret "maas-api-e2e-ipp-b-a1e5d5-serving-cert" not found Oct 30 03:23:06.606661 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:23:06.606627 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls podName:8c6d6950-7765-45f9-b233-c7351302d7a9 nodeName:}" failed. No retries permitted until 2026-10-30 03:23:07.106604614 +0000 UTC m=+1211.108957794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls") pod "maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" (UID: "8c6d6950-7765-45f9-b233-c7351302d7a9") : secret "maas-api-e2e-ipp-b-a1e5d5-serving-cert" not found Oct 30 03:23:06.607260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.607235 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d6950-7765-45f9-b233-c7351302d7a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.609044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.609021 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-metrics-certs\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.631397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.631302 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s55v\" (UniqueName: \"kubernetes.io/projected/8c6d6950-7765-45f9-b233-c7351302d7a9-kube-api-access-8s55v\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:06.656979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.656935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:06.984595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:06.984498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:07.112796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:07.112732 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:07.115211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:07.115184 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:07.269500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:07.269469 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:07.640961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:07.640930 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8"] Oct 30 03:23:07.658923 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:23:07.658644 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6d6950_7765_45f9_b233_c7351302d7a9.slice/crio-6b9a25332e6772a9f49a6ae84055c93cc7366a5c2c0f599e6c4e920595b5d482 WatchSource:0}: Error finding container 6b9a25332e6772a9f49a6ae84055c93cc7366a5c2c0f599e6c4e920595b5d482: Status 404 returned error can't find the container with id 6b9a25332e6772a9f49a6ae84055c93cc7366a5c2c0f599e6c4e920595b5d482 Oct 30 03:23:07.679837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:07.679811 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:23:07.835382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:07.835348 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" event={"ID":"8c6d6950-7765-45f9-b233-c7351302d7a9","Type":"ContainerStarted","Data":"6b9a25332e6772a9f49a6ae84055c93cc7366a5c2c0f599e6c4e920595b5d482"} Oct 30 03:23:08.332294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:08.332267 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:08.844383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:08.844347 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" event={"ID":"8c6d6950-7765-45f9-b233-c7351302d7a9","Type":"ContainerStarted","Data":"2139f7903649becda3d8ea26e92eed964b5bf476e2bdcb709f61ad80f1a485b6"} Oct 30 03:23:08.844561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:08.844423 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:08.884373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:08.884299 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" podStartSLOduration=2.606841423 podStartE2EDuration="2.884285021s" totalImagesPullingTime="277.443598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:08.882534657 +0000 UTC m=+1212.884887835" watchObservedRunningTime="2026-10-30 03:23:08.884285021 +0000 UTC m=+1212.886638209" Oct 30 03:23:10.229158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:10.229094 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:10.332218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:10.332184 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:10.431183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:10.431146 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:11.923318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:11.923281 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:12.034126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:12.034073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:12.963285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:12.963248 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:13.814211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:13.814184 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:14.679070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:14.679037 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:14.855599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:14.855569 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:23:16.418862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:16.418815 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:16.544000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:16.543957 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:16.638436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:16.638403 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:16.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:16.739288 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:18.315780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:18.315434 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:20.035669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.035633 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst"] Oct 30 03:23:20.043025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.042986 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.059046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.059022 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst"] Oct 30 03:23:20.072791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.072739 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6c106c-serving-cert\"" Oct 30 03:23:20.116177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.116147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:20.207658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.207623 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:20.222248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.222218 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcqx\" (UniqueName: \"kubernetes.io/projected/0ac811d5-6d49-4db7-a098-343417e96c71-kube-api-access-chcqx\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.222404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.222343 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac811d5-6d49-4db7-a098-343417e96c71-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.222456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.222411 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.222456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.222441 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-metrics-certs\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.306218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.306122 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:20.323141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.323115 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcqx\" (UniqueName: \"kubernetes.io/projected/0ac811d5-6d49-4db7-a098-343417e96c71-kube-api-access-chcqx\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.323304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.323172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac811d5-6d49-4db7-a098-343417e96c71-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.323304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.323208 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.323304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.323236 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-metrics-certs\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.323823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.323802 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac811d5-6d49-4db7-a098-343417e96c71-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.325746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.325721 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-metrics-certs\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.325844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.325741 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.340876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.340857 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcqx\" (UniqueName: \"kubernetes.io/projected/0ac811d5-6d49-4db7-a098-343417e96c71-kube-api-access-chcqx\") pod \"maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.357674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.357656 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:20.691777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.682564 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst"] Oct 30 03:23:20.924171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:20.924141 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" event={"ID":"0ac811d5-6d49-4db7-a098-343417e96c71","Type":"ContainerStarted","Data":"2dc21d03d684aa1e7075d82c6c4a8d0fea100f7bbd53d5aa7750b2fa59fd70f9"} Oct 30 03:23:21.931279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:21.931243 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" event={"ID":"0ac811d5-6d49-4db7-a098-343417e96c71","Type":"ContainerStarted","Data":"1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3"} Oct 30 03:23:21.931722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:21.931335 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:21.969179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:21.969128 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" podStartSLOduration=1.5417508949999998 podStartE2EDuration="1.969114182s" totalImagesPullingTime="427.363287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:21.967409284 +0000 UTC m=+1225.969762481" watchObservedRunningTime="2026-10-30 03:23:21.969114182 +0000 UTC m=+1225.971467378" Oct 30 03:23:22.195840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:22.195287 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:24.554976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:24.554940 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:24.859889 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:23:24.859858 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:23:26.216875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:26.216842 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:27.909979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:27.909941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:27.941735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:27.941706 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:23:28.986972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:28.986941 4442 generic.go:356] "Generic (PLEG): container finished" podID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerID="2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58" exitCode=0 Oct 30 03:23:28.987517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:28.987021 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" event={"ID":"904c0e21-cf94-46fd-8cfc-e4edaf0105b3","Type":"ContainerDied","Data":"2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58"} Oct 30 03:23:28.987517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:28.987367 4442 scope.go:122] "RemoveContainer" containerID="2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58" Oct 30 03:23:28.988707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:28.988681 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:23:28.988803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:28.988789 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:23:28.988862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:28.988826 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:23:29.627791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:29.624999 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:30.029043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:30.029004 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" event={"ID":"904c0e21-cf94-46fd-8cfc-e4edaf0105b3","Type":"ContainerStarted","Data":"b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0"} Oct 30 03:23:30.351899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:30.351816 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:23:30.353201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:30.353174 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 03:23:30.353352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:30.353231 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 03:23:30.967805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:30.959140 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:23:31.352204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.352170 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 03:23:31.352611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.352231 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 03:23:31.371560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.371526 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d"] Oct 30 03:23:31.375541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.375519 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.396532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.396509 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e0f003-serving-cert\"" Oct 30 03:23:31.420407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.420370 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d"] Oct 30 03:23:31.464535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.464508 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.464712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.464666 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7t8d\" (UniqueName: \"kubernetes.io/projected/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-kube-api-access-l7t8d\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.464712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.464700 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.464903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.464880 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.469430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.469405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:31.566333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.566296 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.566545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.566356 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.566545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.566426 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7t8d\" (UniqueName: \"kubernetes.io/projected/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-kube-api-access-l7t8d\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.566545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.566448 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.567125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.567088 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.569085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.569064 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.569308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.569291 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.609827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.595664 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:31.609827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.605457 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7t8d\" (UniqueName: \"kubernetes.io/projected/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-kube-api-access-l7t8d\") pod \"maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:31.719273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:31.716716 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:32.068836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:32.068812 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d"] Oct 30 03:23:32.069886 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:23:32.069843 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba39eee_d3b9_49d2_bb1d_ff061c53e0ad.slice/crio-3ffb27a127e9d5062b3c21b9bddbc76ccd520ba2a9a03caa241fdd0a7bdf35a6 WatchSource:0}: Error finding container 3ffb27a127e9d5062b3c21b9bddbc76ccd520ba2a9a03caa241fdd0a7bdf35a6: Status 404 returned error can't find the container with id 3ffb27a127e9d5062b3c21b9bddbc76ccd520ba2a9a03caa241fdd0a7bdf35a6 Oct 30 03:23:32.352482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:32.352402 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 03:23:32.352482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:32.352454 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 03:23:32.708734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:32.707860 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:23:33.046533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:33.046497 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" event={"ID":"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad","Type":"ContainerStarted","Data":"5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4"} Oct 30 03:23:33.046694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:33.046564 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:33.046694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:33.046588 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" event={"ID":"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad","Type":"ContainerStarted","Data":"3ffb27a127e9d5062b3c21b9bddbc76ccd520ba2a9a03caa241fdd0a7bdf35a6"} Oct 30 03:23:33.097779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:33.096264 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" podStartSLOduration=1.6310416810000001 podStartE2EDuration="2.096245151s" totalImagesPullingTime="465.20347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:33.096142714 +0000 UTC m=+1237.098495912" watchObservedRunningTime="2026-10-30 03:23:33.096245151 +0000 UTC m=+1237.098598349" Oct 30 03:23:33.352631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:33.352549 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 03:23:33.352631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:33.352599 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 03:23:33.544347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:33.544312 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:34.352923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:34.352886 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 03:23:34.353328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:34.352938 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 03:23:34.921974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:34.921936 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm"] Oct 30 03:23:35.049826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:35.049793 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:35.143701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:35.143652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:35.243685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:35.243600 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:35.352420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:35.352385 4442 patch_prober.go:29] interesting pod/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 03:23:35.352612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:35.352455 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 03:23:36.001826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:36.001791 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:36.356282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:36.356204 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:23:36.357189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:36.357165 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:23:37.289689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:37.289657 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:37.465942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:37.465895 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:38.424848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:38.424813 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:39.057629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:39.057600 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:23:39.922631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:39.922594 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" podUID="f07b0298-6579-4261-af0e-29ef71b97b91" containerName="payload-processing" containerID="cri-o://e5528332105cebddca414e34280e4680fadb4f865ddb13d1538a96773498bbe4" gracePeriod=25 Oct 30 03:23:40.130449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.130410 4442 generic.go:356] "Generic (PLEG): container finished" podID="f07b0298-6579-4261-af0e-29ef71b97b91" containerID="e5528332105cebddca414e34280e4680fadb4f865ddb13d1538a96773498bbe4" exitCode=0 Oct 30 03:23:40.130662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.130486 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" event={"ID":"f07b0298-6579-4261-af0e-29ef71b97b91","Type":"ContainerDied","Data":"e5528332105cebddca414e34280e4680fadb4f865ddb13d1538a96773498bbe4"} Oct 30 03:23:40.180877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.180827 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:23:40.213369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.213341 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:40.319197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.319166 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07b0298-6579-4261-af0e-29ef71b97b91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b0298-6579-4261-af0e-29ef71b97b91-plugins-config-volume\") pod \"f07b0298-6579-4261-af0e-29ef71b97b91\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " Oct 30 03:23:40.319399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.319322 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07b0298-6579-4261-af0e-29ef71b97b91-kube-api-access-4rnnp\" (UniqueName: \"kubernetes.io/projected/f07b0298-6579-4261-af0e-29ef71b97b91-kube-api-access-4rnnp\") pod \"f07b0298-6579-4261-af0e-29ef71b97b91\" (UID: \"f07b0298-6579-4261-af0e-29ef71b97b91\") " Oct 30 03:23:40.319603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.319580 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07b0298-6579-4261-af0e-29ef71b97b91-plugins-config-volume" pod "f07b0298-6579-4261-af0e-29ef71b97b91" (UID: "f07b0298-6579-4261-af0e-29ef71b97b91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:23:40.319789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.319770 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f07b0298-6579-4261-af0e-29ef71b97b91-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:23:40.321598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.321567 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07b0298-6579-4261-af0e-29ef71b97b91-kube-api-access-4rnnp" pod "f07b0298-6579-4261-af0e-29ef71b97b91" (UID: "f07b0298-6579-4261-af0e-29ef71b97b91"). InnerVolumeSpecName "kube-api-access-4rnnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:23:40.420448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:40.420409 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rnnp\" (UniqueName: \"kubernetes.io/projected/f07b0298-6579-4261-af0e-29ef71b97b91-kube-api-access-4rnnp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:23:41.136984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:41.136951 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" Oct 30 03:23:41.136984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:41.136970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm" event={"ID":"f07b0298-6579-4261-af0e-29ef71b97b91","Type":"ContainerDied","Data":"3ee06cac9ad9f58630d1cf80cb421291af39de328c0539240c4b7418389a9056"} Oct 30 03:23:41.137509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:41.137029 4442 scope.go:122] "RemoveContainer" containerID="e5528332105cebddca414e34280e4680fadb4f865ddb13d1538a96773498bbe4" Oct 30 03:23:41.182831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:41.182801 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm"] Oct 30 03:23:41.193436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:41.193396 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a1e5d5-78ddfb557b-mb5sm"] Oct 30 03:23:41.813455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:41.813413 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:42.107473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:42.107395 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07b0298-6579-4261-af0e-29ef71b97b91" path="/var/lib/kubelet/pods/f07b0298-6579-4261-af0e-29ef71b97b91/volumes" Oct 30 03:23:43.405398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:43.405341 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:44.565956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:44.565914 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:45.100330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:45.100292 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:46.234828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:46.234795 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs"] Oct 30 03:23:46.315137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:46.315108 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:46.447298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:46.447225 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:47.415978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:47.415942 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:47.522488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:47.522445 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:23:49.173858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:49.173821 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:50.877068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:50.877039 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:51.235400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.235311 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" podUID="8d03adf8-ce28-414e-8126-4a495919ca0b" containerName="payload-processing" containerID="cri-o://02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5" gracePeriod=25 Oct 30 03:23:51.501808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.501787 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:51.563739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.563704 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d03adf8-ce28-414e-8126-4a495919ca0b-kube-api-access-f29b4\" (UniqueName: \"kubernetes.io/projected/8d03adf8-ce28-414e-8126-4a495919ca0b-kube-api-access-f29b4\") pod \"8d03adf8-ce28-414e-8126-4a495919ca0b\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " Oct 30 03:23:51.563950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.563837 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d03adf8-ce28-414e-8126-4a495919ca0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d03adf8-ce28-414e-8126-4a495919ca0b-plugins-config-volume\") pod \"8d03adf8-ce28-414e-8126-4a495919ca0b\" (UID: \"8d03adf8-ce28-414e-8126-4a495919ca0b\") " Oct 30 03:23:51.564204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.564178 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d03adf8-ce28-414e-8126-4a495919ca0b-plugins-config-volume" pod "8d03adf8-ce28-414e-8126-4a495919ca0b" (UID: "8d03adf8-ce28-414e-8126-4a495919ca0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:23:51.566057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.566027 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d03adf8-ce28-414e-8126-4a495919ca0b-kube-api-access-f29b4" pod "8d03adf8-ce28-414e-8126-4a495919ca0b" (UID: "8d03adf8-ce28-414e-8126-4a495919ca0b"). InnerVolumeSpecName "kube-api-access-f29b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:23:51.665052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.665017 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f29b4\" (UniqueName: \"kubernetes.io/projected/8d03adf8-ce28-414e-8126-4a495919ca0b-kube-api-access-f29b4\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:23:51.665052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:51.665048 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d03adf8-ce28-414e-8126-4a495919ca0b-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:23:52.054020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.053992 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:52.230924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.230890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:52.258978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.258945 4442 generic.go:356] "Generic (PLEG): container finished" podID="8d03adf8-ce28-414e-8126-4a495919ca0b" containerID="02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5" exitCode=0 Oct 30 03:23:52.259234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.259051 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" event={"ID":"8d03adf8-ce28-414e-8126-4a495919ca0b","Type":"ContainerDied","Data":"02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5"} Oct 30 03:23:52.259234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.259063 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" Oct 30 03:23:52.259234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.259101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs" event={"ID":"8d03adf8-ce28-414e-8126-4a495919ca0b","Type":"ContainerDied","Data":"d0ca419387c82e569430cc9ad0da611141ab897838f22903a019c9afda7a5bb4"} Oct 30 03:23:52.259234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.259127 4442 scope.go:122] "RemoveContainer" containerID="02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5" Oct 30 03:23:52.271081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.271059 4442 scope.go:122] "RemoveContainer" containerID="02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5" Oct 30 03:23:52.271390 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:23:52.271370 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5\": container with ID starting with 02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5 not found: ID does not exist" containerID="02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5" Oct 30 03:23:52.271458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.271403 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5"} err="failed to get container status \"02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5\": rpc error: code = NotFound desc = could not find container \"02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5\": container with ID starting with 02d9dcd88af2a727b2fbdec259d0e5779b88a82b577fa634f5a1e92359e08bc5 not found: ID does not exist" Oct 30 03:23:52.305833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.305723 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs"] Oct 30 03:23:52.316096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:52.316073 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6c106c-6ccfc8f874-mrkvs"] Oct 30 03:23:53.229449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:53.229419 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:53.625332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:53.625295 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:54.109535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:54.109505 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d03adf8-ce28-414e-8126-4a495919ca0b" path="/var/lib/kubelet/pods/8d03adf8-ce28-414e-8126-4a495919ca0b/volumes" Oct 30 03:23:54.260372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:54.260337 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:54.489888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:54.489811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:54.861962 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:23:54.861926 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:23:54.942145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:54.942109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:56.957844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:56.957806 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:57.573292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:57.573263 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:57.799556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:57.799510 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:59.239627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.239594 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:59.346619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.346590 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:23:59.392108 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.392076 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm"] Oct 30 03:23:59.392587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.392574 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f07b0298-6579-4261-af0e-29ef71b97b91" containerName="payload-processing" Oct 30 03:23:59.392630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.392589 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07b0298-6579-4261-af0e-29ef71b97b91" containerName="payload-processing" Oct 30 03:23:59.392630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.392614 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8d03adf8-ce28-414e-8126-4a495919ca0b" containerName="payload-processing" Oct 30 03:23:59.392630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.392619 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d03adf8-ce28-414e-8126-4a495919ca0b" containerName="payload-processing" Oct 30 03:23:59.392728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.392688 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07b0298-6579-4261-af0e-29ef71b97b91" containerName="payload-processing" Oct 30 03:23:59.392728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.392698 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d03adf8-ce28-414e-8126-4a495919ca0b" containerName="payload-processing" Oct 30 03:23:59.397544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.397524 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:23:59.413491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.413441 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm"] Oct 30 03:23:59.525344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.525307 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b03a5e\"" Oct 30 03:23:59.525860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.525824 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b03a5e-dockercfg-5rwkl\"" Oct 30 03:23:59.532897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.532868 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlsw\" (UniqueName: \"kubernetes.io/projected/1d1cd6b2-7eed-47d7-813c-96647c8949af-kube-api-access-rmlsw\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:23:59.549918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.549866 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1cd6b2-7eed-47d7-813c-96647c8949af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:23:59.652264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.652215 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1cd6b2-7eed-47d7-813c-96647c8949af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:23:59.652448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.652322 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlsw\" (UniqueName: \"kubernetes.io/projected/1d1cd6b2-7eed-47d7-813c-96647c8949af-kube-api-access-rmlsw\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:23:59.652921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.652900 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1cd6b2-7eed-47d7-813c-96647c8949af-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:23:59.670309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.670281 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlsw\" (UniqueName: \"kubernetes.io/projected/1d1cd6b2-7eed-47d7-813c-96647c8949af-kube-api-access-rmlsw\") pod \"payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:23:59.712025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:23:59.711984 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:24:00.014598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:00.014568 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm"] Oct 30 03:24:00.016096 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:24:00.016066 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1cd6b2_7eed_47d7_813c_96647c8949af.slice/crio-d83a8f524be4b4748b7c237c4f7cb784f5baa3ee2a4250dbe859d6df84838ae2 WatchSource:0}: Error finding container d83a8f524be4b4748b7c237c4f7cb784f5baa3ee2a4250dbe859d6df84838ae2: Status 404 returned error can't find the container with id d83a8f524be4b4748b7c237c4f7cb784f5baa3ee2a4250dbe859d6df84838ae2 Oct 30 03:24:00.339925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:00.339830 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" event={"ID":"1d1cd6b2-7eed-47d7-813c-96647c8949af","Type":"ContainerStarted","Data":"521c147bb43c580eedbc9ff2470d74664d20f098589125221473f968ed8fde3c"} Oct 30 03:24:00.339925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:00.339889 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" event={"ID":"1d1cd6b2-7eed-47d7-813c-96647c8949af","Type":"ContainerStarted","Data":"d83a8f524be4b4748b7c237c4f7cb784f5baa3ee2a4250dbe859d6df84838ae2"} Oct 30 03:24:00.339925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:00.339914 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:24:00.375974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:00.375921 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" podStartSLOduration=1.375905687 podStartE2EDuration="1.375905687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:00.375704602 +0000 UTC m=+1264.378057801" watchObservedRunningTime="2026-10-30 03:24:00.375905687 +0000 UTC m=+1264.378258885" Oct 30 03:24:01.304187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:01.304159 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:01.437863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:01.437818 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:01.562967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:01.562887 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:01.672910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:01.672883 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:03.374801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:03.374770 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:05.367988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:05.367956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:06.924441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:06.924405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:08.206416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:08.206382 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:08.394271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:08.394236 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:08.669551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:08.669509 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:09.358715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:09.358679 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:11.348443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:11.348415 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:24:11.502205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:11.502175 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:13.213223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:13.213185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:14.126794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:14.126736 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:14.573077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:14.573045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:14.680829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:14.680802 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:14.958465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:14.958380 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:15.054938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:15.054905 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:17.230368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:17.230335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:17.652029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:17.651991 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:17.752452 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:17.752420 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:19.117146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.117110 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:19.373393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.373307 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:19.495217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.495182 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:19.619252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.619220 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn"] Oct 30 03:24:19.624425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.624258 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:19.651493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.651449 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-dff241-dockercfg-zn7jq\"" Oct 30 03:24:19.651671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.651517 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-dff241\"" Oct 30 03:24:19.663062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.663032 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn"] Oct 30 03:24:19.719541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.719510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mss6x\" (UniqueName: \"kubernetes.io/projected/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-kube-api-access-mss6x\") pod \"payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:19.719541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.719546 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:19.821256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.821224 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:19.821422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.821387 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mss6x\" (UniqueName: \"kubernetes.io/projected/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-kube-api-access-mss6x\") pod \"payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:19.821893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.821875 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:19.839260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.839230 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mss6x\" (UniqueName: \"kubernetes.io/projected/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-kube-api-access-mss6x\") pod \"payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:19.972253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:19.972149 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:20.302741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:20.302712 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn"] Oct 30 03:24:20.303212 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:24:20.303191 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc4aa7e_7ba9_4e29_bf10_d47d1c5ab3f9.slice/crio-bb490b929cff2389725e85409f1b1b1fcac611b34066b806643d0cffe941bf2f WatchSource:0}: Error finding container bb490b929cff2389725e85409f1b1b1fcac611b34066b806643d0cffe941bf2f: Status 404 returned error can't find the container with id bb490b929cff2389725e85409f1b1b1fcac611b34066b806643d0cffe941bf2f Oct 30 03:24:20.544942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:20.533457 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" event={"ID":"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9","Type":"ContainerStarted","Data":"bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c"} Oct 30 03:24:20.544942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:20.533536 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" event={"ID":"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9","Type":"ContainerStarted","Data":"bb490b929cff2389725e85409f1b1b1fcac611b34066b806643d0cffe941bf2f"} Oct 30 03:24:20.544942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:20.534480 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:20.574501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:20.573042 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" podStartSLOduration=1.573025317 podStartE2EDuration="1.573025317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:20.570521775 +0000 UTC m=+1284.572874972" watchObservedRunningTime="2026-10-30 03:24:20.573025317 +0000 UTC m=+1284.575378513" Oct 30 03:24:20.604604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:20.604574 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:20.780605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:20.780575 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:21.336697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.336668 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78"] Oct 30 03:24:21.340586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.340565 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.378583 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.378548 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78"] Oct 30 03:24:21.419254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.419227 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e0f003-dockercfg-xb8r7\"" Oct 30 03:24:21.420653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.420632 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e0f003\"" Oct 30 03:24:21.442105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.442076 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjrf\" (UniqueName: \"kubernetes.io/projected/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-kube-api-access-tjjrf\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.442271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.442248 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.543284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.543252 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjrf\" (UniqueName: \"kubernetes.io/projected/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-kube-api-access-tjjrf\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.543456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.543386 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.544067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.544049 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.561318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.561285 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjrf\" (UniqueName: \"kubernetes.io/projected/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-kube-api-access-tjjrf\") pod \"payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.654377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.654299 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:21.902177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.902139 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2"] Oct 30 03:24:21.909601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.909535 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:21.922274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.922107 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2"] Oct 30 03:24:21.933996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.933968 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b03a5e-serving-cert\"" Oct 30 03:24:21.948723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.948698 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:21.948873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.948732 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadf6862-cb6d-449d-833e-67ccbb64cb19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:21.948917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.948880 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-metrics-certs\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:21.948968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.948930 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkvp\" (UniqueName: \"kubernetes.io/projected/eadf6862-cb6d-449d-833e-67ccbb64cb19-kube-api-access-ntkvp\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:21.961278 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.961203 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78"] Oct 30 03:24:21.964556 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:24:21.964526 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c43e5a5_5519_45ea_b3ff_4cd2f8a6daf6.slice/crio-95ab87f0d5bf18c0161930c0bef7c90d553416732539eda114dd7fb6d9f3840a WatchSource:0}: Error finding container 95ab87f0d5bf18c0161930c0bef7c90d553416732539eda114dd7fb6d9f3840a: Status 404 returned error can't find the container with id 95ab87f0d5bf18c0161930c0bef7c90d553416732539eda114dd7fb6d9f3840a Oct 30 03:24:21.980623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:21.980597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:22.049987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.049953 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.049987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.049988 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadf6862-cb6d-449d-833e-67ccbb64cb19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.050175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.050039 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-metrics-certs\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.050175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.050074 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkvp\" (UniqueName: \"kubernetes.io/projected/eadf6862-cb6d-449d-833e-67ccbb64cb19-kube-api-access-ntkvp\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.050710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.050687 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadf6862-cb6d-449d-833e-67ccbb64cb19-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.052474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.052449 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-metrics-certs\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.052560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.052525 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.068032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.068011 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkvp\" (UniqueName: \"kubernetes.io/projected/eadf6862-cb6d-449d-833e-67ccbb64cb19-kube-api-access-ntkvp\") pod \"maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.083772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.083722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:22.182502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.182424 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:22.229783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.229425 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:22.546017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.545985 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" event={"ID":"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6","Type":"ContainerStarted","Data":"511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f"} Oct 30 03:24:22.546491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.546034 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" event={"ID":"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6","Type":"ContainerStarted","Data":"95ab87f0d5bf18c0161930c0bef7c90d553416732539eda114dd7fb6d9f3840a"} Oct 30 03:24:22.546491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.546208 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:22.560897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.560876 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2"] Oct 30 03:24:22.562228 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:24:22.562201 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeadf6862_cb6d_449d_833e_67ccbb64cb19.slice/crio-377fc49710ec5c7fafa40d43df9a125fc0107fd8ed43f29c36d204566e70ddac WatchSource:0}: Error finding container 377fc49710ec5c7fafa40d43df9a125fc0107fd8ed43f29c36d204566e70ddac: Status 404 returned error can't find the container with id 377fc49710ec5c7fafa40d43df9a125fc0107fd8ed43f29c36d204566e70ddac Oct 30 03:24:22.563873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.563855 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:24:22.591311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:22.591271 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" podStartSLOduration=1.5912549139999999 podStartE2EDuration="1.591254914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:22.590108916 +0000 UTC m=+1286.592462114" watchObservedRunningTime="2026-10-30 03:24:22.591254914 +0000 UTC m=+1286.593608111" Oct 30 03:24:23.404230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:23.404196 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:23.553522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:23.553484 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" event={"ID":"eadf6862-cb6d-449d-833e-67ccbb64cb19","Type":"ContainerStarted","Data":"d7438c57a6fe79c8922dff7d9ddd83cc8d3ff070eab1a9aee0d835ea3bcf64fc"} Oct 30 03:24:23.554044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:23.553552 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" event={"ID":"eadf6862-cb6d-449d-833e-67ccbb64cb19","Type":"ContainerStarted","Data":"377fc49710ec5c7fafa40d43df9a125fc0107fd8ed43f29c36d204566e70ddac"} Oct 30 03:24:23.554044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:23.553669 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:23.593222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:23.593173 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" podStartSLOduration=2.254080753 podStartE2EDuration="2.593159723s" totalImagesPullingTime="339.07897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:23.59010626 +0000 UTC m=+1287.592459461" watchObservedRunningTime="2026-10-30 03:24:23.593159723 +0000 UTC m=+1287.595512920" Oct 30 03:24:23.865545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:23.865510 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:23.977730 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:23.977698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:24.864197 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:24:24.864159 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:24:26.163984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:26.163920 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:27.981022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:27.980993 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:28.109638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:28.109604 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:28.755421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:28.755393 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:29.496797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.496768 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:29.564595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.564562 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:24:29.593105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.593069 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf"] Oct 30 03:24:29.597345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.597323 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:29.616438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.616414 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf"] Oct 30 03:24:29.620006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.619983 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efbbc5\"" Oct 30 03:24:29.620315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.620296 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efbbc5-dockercfg-jtp2k\"" Oct 30 03:24:29.647148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.647107 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c17810-ebe5-41df-91e3-375e080d5e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:29.647321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.647169 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps4b\" (UniqueName: \"kubernetes.io/projected/30c17810-ebe5-41df-91e3-375e080d5e4d-kube-api-access-pps4b\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:29.656183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.656160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:29.767239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.753556 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c17810-ebe5-41df-91e3-375e080d5e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:29.767239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.753629 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pps4b\" (UniqueName: \"kubernetes.io/projected/30c17810-ebe5-41df-91e3-375e080d5e4d-kube-api-access-pps4b\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:29.767239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.756337 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c17810-ebe5-41df-91e3-375e080d5e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:29.808327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.808298 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps4b\" (UniqueName: \"kubernetes.io/projected/30c17810-ebe5-41df-91e3-375e080d5e4d-kube-api-access-pps4b\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:29.912202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:29.912177 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:30.231019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:30.230994 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf"] Oct 30 03:24:30.232426 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:24:30.232396 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c17810_ebe5_41df_91e3_375e080d5e4d.slice/crio-e6576c442f33c42b37105428463e1215e24cf1f95de2b491b3193684e623b33c WatchSource:0}: Error finding container e6576c442f33c42b37105428463e1215e24cf1f95de2b491b3193684e623b33c: Status 404 returned error can't find the container with id e6576c442f33c42b37105428463e1215e24cf1f95de2b491b3193684e623b33c Oct 30 03:24:30.611943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:30.609465 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" event={"ID":"30c17810-ebe5-41df-91e3-375e080d5e4d","Type":"ContainerStarted","Data":"c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d"} Oct 30 03:24:30.611943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:30.609570 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" event={"ID":"30c17810-ebe5-41df-91e3-375e080d5e4d","Type":"ContainerStarted","Data":"e6576c442f33c42b37105428463e1215e24cf1f95de2b491b3193684e623b33c"} Oct 30 03:24:30.611943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:30.610665 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:30.658796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:30.658719 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" podStartSLOduration=1.658703765 podStartE2EDuration="1.658703765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:30.656698613 +0000 UTC m=+1294.659051811" watchObservedRunningTime="2026-10-30 03:24:30.658703765 +0000 UTC m=+1294.661056962" Oct 30 03:24:30.662972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:30.662935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:32.248523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:32.248481 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:32.549484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:32.549450 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:24:33.556367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:33.556339 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:24:34.056867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:34.056837 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:35.874904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:35.874872 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6"] Oct 30 03:24:35.875387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:35.875093 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" podUID="1ae1db7c-ff63-4348-b529-18f27b7a5ccb" containerName="maas-api" containerID="cri-o://9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d" gracePeriod=30 Oct 30 03:24:35.951201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:35.951171 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:36.028964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.028934 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74"] Oct 30 03:24:36.029188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.029164 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" podUID="26770af4-ea80-49a5-85c1-a5b1e30883b1" containerName="payload-processing" containerID="cri-o://0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9" gracePeriod=30 Oct 30 03:24:36.115986 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.115955 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:36.172509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.172483 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:24:36.229316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.229289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:36.248741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.248712 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-openshift-service-ca\") pod \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " Oct 30 03:24:36.248936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.248813 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-metrics-certs\") pod \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " Oct 30 03:24:36.248936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.248924 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls\") pod \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " Oct 30 03:24:36.249144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.249117 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-openshift-service-ca" pod "1ae1db7c-ff63-4348-b529-18f27b7a5ccb" (UID: "1ae1db7c-ff63-4348-b529-18f27b7a5ccb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:24:36.249359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.249333 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:36.251856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.251829 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls" pod "1ae1db7c-ff63-4348-b529-18f27b7a5ccb" (UID: "1ae1db7c-ff63-4348-b529-18f27b7a5ccb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:24:36.251946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.251868 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-metrics-certs" pod "1ae1db7c-ff63-4348-b529-18f27b7a5ccb" (UID: "1ae1db7c-ff63-4348-b529-18f27b7a5ccb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:24:36.279885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.279859 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:24:36.426510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.426426 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26770af4-ea80-49a5-85c1-a5b1e30883b1-kube-api-access-2k9hf\" (UniqueName: \"kubernetes.io/projected/26770af4-ea80-49a5-85c1-a5b1e30883b1-kube-api-access-2k9hf\") pod \"26770af4-ea80-49a5-85c1-a5b1e30883b1\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " Oct 30 03:24:36.426921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.426892 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26770af4-ea80-49a5-85c1-a5b1e30883b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26770af4-ea80-49a5-85c1-a5b1e30883b1-plugins-config-volume\") pod \"26770af4-ea80-49a5-85c1-a5b1e30883b1\" (UID: \"26770af4-ea80-49a5-85c1-a5b1e30883b1\") " Oct 30 03:24:36.427097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.427083 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-kube-api-access-2mxn2\" (UniqueName: \"kubernetes.io/projected/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-kube-api-access-2mxn2\") pod \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\" (UID: \"1ae1db7c-ff63-4348-b529-18f27b7a5ccb\") " Oct 30 03:24:36.427577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.427560 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:36.427671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.427663 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:36.429815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.428962 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26770af4-ea80-49a5-85c1-a5b1e30883b1-plugins-config-volume" pod "26770af4-ea80-49a5-85c1-a5b1e30883b1" (UID: "26770af4-ea80-49a5-85c1-a5b1e30883b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:24:36.434878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.434846 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26770af4-ea80-49a5-85c1-a5b1e30883b1-kube-api-access-2k9hf" pod "26770af4-ea80-49a5-85c1-a5b1e30883b1" (UID: "26770af4-ea80-49a5-85c1-a5b1e30883b1"). InnerVolumeSpecName "kube-api-access-2k9hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:24:36.435168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.435114 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-kube-api-access-2mxn2" pod "1ae1db7c-ff63-4348-b529-18f27b7a5ccb" (UID: "1ae1db7c-ff63-4348-b529-18f27b7a5ccb"). InnerVolumeSpecName "kube-api-access-2mxn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:24:36.528944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.528912 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26770af4-ea80-49a5-85c1-a5b1e30883b1-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:36.528944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.528938 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mxn2\" (UniqueName: \"kubernetes.io/projected/1ae1db7c-ff63-4348-b529-18f27b7a5ccb-kube-api-access-2mxn2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:36.528944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.528947 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9hf\" (UniqueName: \"kubernetes.io/projected/26770af4-ea80-49a5-85c1-a5b1e30883b1-kube-api-access-2k9hf\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:36.645168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.645131 4442 generic.go:356] "Generic (PLEG): container finished" podID="26770af4-ea80-49a5-85c1-a5b1e30883b1" containerID="0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9" exitCode=0 Oct 30 03:24:36.645337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.645183 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" Oct 30 03:24:36.645337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.645226 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" event={"ID":"26770af4-ea80-49a5-85c1-a5b1e30883b1","Type":"ContainerDied","Data":"0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9"} Oct 30 03:24:36.645337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.645278 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74" event={"ID":"26770af4-ea80-49a5-85c1-a5b1e30883b1","Type":"ContainerDied","Data":"b72285c7370c1c1b6c806b204bf4e76dd3b0f5c6f1d6c177002523c8288f71be"} Oct 30 03:24:36.645337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.645306 4442 scope.go:122] "RemoveContainer" containerID="0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9" Oct 30 03:24:36.646733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.646707 4442 generic.go:356] "Generic (PLEG): container finished" podID="1ae1db7c-ff63-4348-b529-18f27b7a5ccb" containerID="9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d" exitCode=0 Oct 30 03:24:36.646909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.646747 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" Oct 30 03:24:36.646909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.646773 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" event={"ID":"1ae1db7c-ff63-4348-b529-18f27b7a5ccb","Type":"ContainerDied","Data":"9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d"} Oct 30 03:24:36.646909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.646807 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6" event={"ID":"1ae1db7c-ff63-4348-b529-18f27b7a5ccb","Type":"ContainerDied","Data":"72940fe0dcd42c173a62b711fc04ca0694547189057f9cfee8c362882f1e3779"} Oct 30 03:24:36.658262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.658238 4442 scope.go:122] "RemoveContainer" containerID="0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9" Oct 30 03:24:36.658849 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:24:36.658828 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9\": container with ID starting with 0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9 not found: ID does not exist" containerID="0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9" Oct 30 03:24:36.658944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.658864 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9"} err="failed to get container status \"0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9\": rpc error: code = NotFound desc = could not find container \"0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9\": container with ID starting with 0770d6ad69e9753b044ca95b52f4adedf8cd8905acdd89433a070a88a8a2c5b9 not found: ID does not exist" Oct 30 03:24:36.658944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.658885 4442 scope.go:122] "RemoveContainer" containerID="9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d" Oct 30 03:24:36.662400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.662378 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:36.668868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.668840 4442 scope.go:122] "RemoveContainer" containerID="9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d" Oct 30 03:24:36.669144 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:24:36.669124 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d\": container with ID starting with 9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d not found: ID does not exist" containerID="9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d" Oct 30 03:24:36.669218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.669153 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d"} err="failed to get container status \"9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d\": rpc error: code = NotFound desc = could not find container \"9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d\": container with ID starting with 9bf77264958e44ad31c17649d2432aeb7baecb9c9a752a5308dab4facf5cbe1d not found: ID does not exist" Oct 30 03:24:36.740360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.740273 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6"] Oct 30 03:24:36.741784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.741720 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d6c536-6c9fb48f46-845t6"] Oct 30 03:24:36.760688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.760654 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74"] Oct 30 03:24:36.763217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:36.763194 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-d6c536-584f8cd944-djv74"] Oct 30 03:24:38.106851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:38.106819 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae1db7c-ff63-4348-b529-18f27b7a5ccb" path="/var/lib/kubelet/pods/1ae1db7c-ff63-4348-b529-18f27b7a5ccb/volumes" Oct 30 03:24:38.107240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:38.107219 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26770af4-ea80-49a5-85c1-a5b1e30883b1" path="/var/lib/kubelet/pods/26770af4-ea80-49a5-85c1-a5b1e30883b1/volumes" Oct 30 03:24:38.262268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:38.262232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:38.438918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:38.437387 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:24:39.551061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:39.550980 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:39.706493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:39.706463 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:39.810533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:39.810453 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:40.188733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:40.188633 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:40.464013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:40.463928 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:40.588422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:40.588381 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:40.853906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:40.853878 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:42.499173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.499138 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9"] Oct 30 03:24:42.499658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.499644 4442 cpu_manager.go:414] "containerMap: removing container" podUID="26770af4-ea80-49a5-85c1-a5b1e30883b1" containerName="payload-processing" Oct 30 03:24:42.499713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.499659 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26770af4-ea80-49a5-85c1-a5b1e30883b1" containerName="payload-processing" Oct 30 03:24:42.499713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.499671 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1ae1db7c-ff63-4348-b529-18f27b7a5ccb" containerName="maas-api" Oct 30 03:24:42.499713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.499676 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae1db7c-ff63-4348-b529-18f27b7a5ccb" containerName="maas-api" Oct 30 03:24:42.499838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.499810 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="26770af4-ea80-49a5-85c1-a5b1e30883b1" containerName="payload-processing" Oct 30 03:24:42.499838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.499819 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae1db7c-ff63-4348-b529-18f27b7a5ccb" containerName="maas-api" Oct 30 03:24:42.504511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.504493 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.521952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.521927 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9"] Oct 30 03:24:42.529915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.529893 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-dff241-serving-cert\"" Oct 30 03:24:42.544702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.544676 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.544850 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.544741 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59e2112-73c9-40a9-8ea6-5913e9f76c39-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.544921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.544895 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqnkk\" (UniqueName: \"kubernetes.io/projected/a59e2112-73c9-40a9-8ea6-5913e9f76c39-kube-api-access-bqnkk\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.545010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.544952 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-metrics-certs\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.581969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.581943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:42.623275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.623253 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:24:42.646598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.646572 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.646689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.646600 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59e2112-73c9-40a9-8ea6-5913e9f76c39-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.646689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.646675 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqnkk\" (UniqueName: \"kubernetes.io/projected/a59e2112-73c9-40a9-8ea6-5913e9f76c39-kube-api-access-bqnkk\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.646822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.646706 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-metrics-certs\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.646822 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:24:42.646719 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-serving-cert: secret "maas-api-e2e-praxis-a-dff241-serving-cert" not found Oct 30 03:24:42.646822 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:24:42.646806 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls podName:a59e2112-73c9-40a9-8ea6-5913e9f76c39 nodeName:}" failed. No retries permitted until 2026-10-30 03:24:43.146786441 +0000 UTC m=+1307.149139632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls") pod "maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" (UID: "a59e2112-73c9-40a9-8ea6-5913e9f76c39") : secret "maas-api-e2e-praxis-a-dff241-serving-cert" not found Oct 30 03:24:42.647291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.647275 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59e2112-73c9-40a9-8ea6-5913e9f76c39-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.649337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.649317 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-metrics-certs\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.665554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.665532 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqnkk\" (UniqueName: \"kubernetes.io/projected/a59e2112-73c9-40a9-8ea6-5913e9f76c39-kube-api-access-bqnkk\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:42.677967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.677944 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:42.897007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:42.896934 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:43.152019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:43.151934 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:43.154661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:43.154631 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:43.417694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:43.417599 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:43.769316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:43.769285 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9"] Oct 30 03:24:43.769824 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:24:43.769795 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59e2112_73c9_40a9_8ea6_5913e9f76c39.slice/crio-5d2dc6b972d5a0012e83db47525b568d9ba28b5400ac24a113a0e596235af900 WatchSource:0}: Error finding container 5d2dc6b972d5a0012e83db47525b568d9ba28b5400ac24a113a0e596235af900: Status 404 returned error can't find the container with id 5d2dc6b972d5a0012e83db47525b568d9ba28b5400ac24a113a0e596235af900 Oct 30 03:24:44.416334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:44.416255 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:44.700725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:44.700645 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:44.715926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:44.715892 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" event={"ID":"a59e2112-73c9-40a9-8ea6-5913e9f76c39","Type":"ContainerStarted","Data":"4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027"} Oct 30 03:24:44.716137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:44.715962 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" event={"ID":"a59e2112-73c9-40a9-8ea6-5913e9f76c39","Type":"ContainerStarted","Data":"5d2dc6b972d5a0012e83db47525b568d9ba28b5400ac24a113a0e596235af900"} Oct 30 03:24:44.716137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:44.716091 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:44.775283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:44.775231 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" podStartSLOduration=2.475874464 podStartE2EDuration="2.775218291s" totalImagesPullingTime="299.343827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:44.772569062 +0000 UTC m=+1308.774922259" watchObservedRunningTime="2026-10-30 03:24:44.775218291 +0000 UTC m=+1308.777571488" Oct 30 03:24:45.818177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:45.818145 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:46.382138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:46.382103 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:46.669114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:46.669028 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:47.783984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:47.783954 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:48.517331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:48.517302 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:49.904698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:49.904662 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt"] Oct 30 03:24:49.905167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:49.904939 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" podUID="d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" containerName="maas-api" containerID="cri-o://ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab" gracePeriod=30 Oct 30 03:24:49.978687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:49.978652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:50.154624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.154593 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:50.185297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.185271 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:24:50.257405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.257375 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-openshift-service-ca\") pod \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " Oct 30 03:24:50.257592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.257470 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls\") pod \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " Oct 30 03:24:50.257592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.257534 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-metrics-certs\") pod \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " Oct 30 03:24:50.257592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.257575 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-kube-api-access-hfz9k\" (UniqueName: \"kubernetes.io/projected/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-kube-api-access-hfz9k\") pod \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\" (UID: \"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479\") " Oct 30 03:24:50.257899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.257846 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-openshift-service-ca" pod "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" (UID: "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:24:50.258984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.258958 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:50.260171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.260127 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls" pod "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" (UID: "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:24:50.260397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.260366 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-metrics-certs" pod "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" (UID: "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:24:50.260482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.260434 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-kube-api-access-hfz9k" pod "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" (UID: "d3af3ca2-1311-4de4-a94f-2fd1b5a5e479"). InnerVolumeSpecName "kube-api-access-hfz9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:24:50.268784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.268716 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:50.360036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.360004 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:50.360036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.360037 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:50.360292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.360049 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfz9k\" (UniqueName: \"kubernetes.io/projected/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479-kube-api-access-hfz9k\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:50.728108 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.728080 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:24:50.753507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.753475 4442 generic.go:356] "Generic (PLEG): container finished" podID="d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" containerID="ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab" exitCode=0 Oct 30 03:24:50.753649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.753523 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" Oct 30 03:24:50.753649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.753563 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" event={"ID":"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479","Type":"ContainerDied","Data":"ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab"} Oct 30 03:24:50.753649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.753615 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt" event={"ID":"d3af3ca2-1311-4de4-a94f-2fd1b5a5e479","Type":"ContainerDied","Data":"cae9dd8a67a17cfc7a6dda6ab3d7ab4104df0963919614e591f75038c7bc29a0"} Oct 30 03:24:50.753649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.753646 4442 scope.go:122] "RemoveContainer" containerID="ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab" Oct 30 03:24:50.764557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.764539 4442 scope.go:122] "RemoveContainer" containerID="ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab" Oct 30 03:24:50.764839 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:24:50.764818 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab\": container with ID starting with ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab not found: ID does not exist" containerID="ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab" Oct 30 03:24:50.764900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.764846 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab"} err="failed to get container status \"ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab\": rpc error: code = NotFound desc = could not find container \"ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab\": container with ID starting with ce6178cf3987b0b83c634341a8cc606ceba74bbb717eadd65a4cc38c721c74ab not found: ID does not exist" Oct 30 03:24:50.809996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.809965 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt"] Oct 30 03:24:50.814189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.814161 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:50.816049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:50.816025 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-04fdf3-64f5f5964d-6fztt"] Oct 30 03:24:52.125189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:52.125152 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" path="/var/lib/kubelet/pods/d3af3ca2-1311-4de4-a94f-2fd1b5a5e479/volumes" Oct 30 03:24:52.533367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:52.533329 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:52.818865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:52.818777 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:53.099777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.099685 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl"] Oct 30 03:24:53.100316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.100302 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" containerName="maas-api" Oct 30 03:24:53.100382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.100319 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" containerName="maas-api" Oct 30 03:24:53.100441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.100409 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3af3ca2-1311-4de4-a94f-2fd1b5a5e479" containerName="maas-api" Oct 30 03:24:53.105095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.105075 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.119697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.119673 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-efbbc5-serving-cert\"" Oct 30 03:24:53.135227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.135101 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl"] Oct 30 03:24:53.207502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.207466 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgvp\" (UniqueName: \"kubernetes.io/projected/e8d319f8-7558-4002-9911-cf2df2061b02-kube-api-access-sdgvp\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.207674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.207574 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.207674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.207639 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d319f8-7558-4002-9911-cf2df2061b02-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.207674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.207664 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-metrics-certs\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.219976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.219950 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:53.308746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.308711 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d319f8-7558-4002-9911-cf2df2061b02-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.308746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.308747 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-metrics-certs\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.308984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.308892 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgvp\" (UniqueName: \"kubernetes.io/projected/e8d319f8-7558-4002-9911-cf2df2061b02-kube-api-access-sdgvp\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.308984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.308967 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.309419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.309398 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d319f8-7558-4002-9911-cf2df2061b02-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.311438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.311417 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-metrics-certs\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.311559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.311517 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-maas-api-tls\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.326569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.326547 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgvp\" (UniqueName: \"kubernetes.io/projected/e8d319f8-7558-4002-9911-cf2df2061b02-kube-api-access-sdgvp\") pod \"maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.356796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.356665 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:53.418547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.418516 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:53.458029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.457996 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:53.628967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.628886 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:53.722238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.722193 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm"] Oct 30 03:24:53.784363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.784334 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl"] Oct 30 03:24:53.817118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.817073 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" event={"ID":"e8d319f8-7558-4002-9911-cf2df2061b02","Type":"ContainerStarted","Data":"9c81168e2d4fff4b30f346425ab8c02f85c7bc94c8e794f5aab373296eca8b9a"} Oct 30 03:24:53.851328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.851290 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:53.955034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:53.954883 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:54.083780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:54.074110 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:54.422609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:54.422575 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:54.824661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:54.824626 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" event={"ID":"e8d319f8-7558-4002-9911-cf2df2061b02","Type":"ContainerStarted","Data":"dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4"} Oct 30 03:24:54.824855 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:54.824690 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:24:54.864041 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:24:54.863999 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:24:54.864232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:54.863998 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" podStartSLOduration=1.5118186260000002 podStartE2EDuration="1.863981018s" totalImagesPullingTime="352.162392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:24:54.862380504 +0000 UTC m=+1318.864733701" watchObservedRunningTime="2026-10-30 03:24:54.863981018 +0000 UTC m=+1318.866334215" Oct 30 03:24:55.157032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:55.156943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:56.900889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:56.900854 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:58.593113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:58.593077 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:24:58.723145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:58.723080 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" podUID="1d1cd6b2-7eed-47d7-813c-96647c8949af" containerName="payload-pre-processing" containerID="cri-o://521c147bb43c580eedbc9ff2470d74664d20f098589125221473f968ed8fde3c" gracePeriod=25 Oct 30 03:24:58.848580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:58.848514 4442 generic.go:356] "Generic (PLEG): container finished" podID="1d1cd6b2-7eed-47d7-813c-96647c8949af" containerID="521c147bb43c580eedbc9ff2470d74664d20f098589125221473f968ed8fde3c" exitCode=0 Oct 30 03:24:58.848580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:58.848560 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" event={"ID":"1d1cd6b2-7eed-47d7-813c-96647c8949af","Type":"ContainerDied","Data":"521c147bb43c580eedbc9ff2470d74664d20f098589125221473f968ed8fde3c"} Oct 30 03:24:58.974841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:58.974819 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:24:59.025317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.025272 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1cd6b2-7eed-47d7-813c-96647c8949af-kube-api-access-rmlsw\" (UniqueName: \"kubernetes.io/projected/1d1cd6b2-7eed-47d7-813c-96647c8949af-kube-api-access-rmlsw\") pod \"1d1cd6b2-7eed-47d7-813c-96647c8949af\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " Oct 30 03:24:59.025617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.025585 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1cd6b2-7eed-47d7-813c-96647c8949af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1cd6b2-7eed-47d7-813c-96647c8949af-plugins-config-volume\") pod \"1d1cd6b2-7eed-47d7-813c-96647c8949af\" (UID: \"1d1cd6b2-7eed-47d7-813c-96647c8949af\") " Oct 30 03:24:59.026217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.026188 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1cd6b2-7eed-47d7-813c-96647c8949af-plugins-config-volume" pod "1d1cd6b2-7eed-47d7-813c-96647c8949af" (UID: "1d1cd6b2-7eed-47d7-813c-96647c8949af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:24:59.028930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.028901 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1cd6b2-7eed-47d7-813c-96647c8949af-kube-api-access-rmlsw" pod "1d1cd6b2-7eed-47d7-813c-96647c8949af" (UID: "1d1cd6b2-7eed-47d7-813c-96647c8949af"). InnerVolumeSpecName "kube-api-access-rmlsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:24:59.127315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.127223 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmlsw\" (UniqueName: \"kubernetes.io/projected/1d1cd6b2-7eed-47d7-813c-96647c8949af-kube-api-access-rmlsw\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:59.127315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.127254 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d1cd6b2-7eed-47d7-813c-96647c8949af-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:24:59.855802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.855769 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" Oct 30 03:24:59.855802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.855793 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm" event={"ID":"1d1cd6b2-7eed-47d7-813c-96647c8949af","Type":"ContainerDied","Data":"d83a8f524be4b4748b7c237c4f7cb784f5baa3ee2a4250dbe859d6df84838ae2"} Oct 30 03:24:59.856260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.855855 4442 scope.go:122] "RemoveContainer" containerID="521c147bb43c580eedbc9ff2470d74664d20f098589125221473f968ed8fde3c" Oct 30 03:24:59.904294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.904262 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm"] Oct 30 03:24:59.914816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:24:59.914788 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b03a5e-57d64dd87-ttdbm"] Oct 30 03:25:00.106988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:00.106910 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1cd6b2-7eed-47d7-813c-96647c8949af" path="/var/lib/kubelet/pods/1d1cd6b2-7eed-47d7-813c-96647c8949af/volumes" Oct 30 03:25:00.352051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:00.352015 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:00.697263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:00.697214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:00.835485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:00.835458 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:25:02.397671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:02.397630 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:02.639378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:02.639332 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:04.322252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:04.322217 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:05.658160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:05.658122 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8"] Oct 30 03:25:05.658638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:05.658376 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" podUID="8c6d6950-7765-45f9-b233-c7351302d7a9" containerName="maas-api" containerID="cri-o://2139f7903649becda3d8ea26e92eed964b5bf476e2bdcb709f61ad80f1a485b6" gracePeriod=30 Oct 30 03:25:05.896818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:05.896742 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" event={"ID":"8c6d6950-7765-45f9-b233-c7351302d7a9","Type":"ContainerDied","Data":"2139f7903649becda3d8ea26e92eed964b5bf476e2bdcb709f61ad80f1a485b6"} Oct 30 03:25:05.897085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:05.896697 4442 generic.go:356] "Generic (PLEG): container finished" podID="8c6d6950-7765-45f9-b233-c7351302d7a9" containerID="2139f7903649becda3d8ea26e92eed964b5bf476e2bdcb709f61ad80f1a485b6" exitCode=0 Oct 30 03:25:05.931885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:05.931857 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:25:05.947091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:05.947069 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:06.007789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.007724 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6d6950-7765-45f9-b233-c7351302d7a9-kube-api-access-8s55v\" (UniqueName: \"kubernetes.io/projected/8c6d6950-7765-45f9-b233-c7351302d7a9-kube-api-access-8s55v\") pod \"8c6d6950-7765-45f9-b233-c7351302d7a9\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " Oct 30 03:25:06.007970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.007865 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6d6950-7765-45f9-b233-c7351302d7a9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d6950-7765-45f9-b233-c7351302d7a9-openshift-service-ca\") pod \"8c6d6950-7765-45f9-b233-c7351302d7a9\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " Oct 30 03:25:06.007970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.007905 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-metrics-certs\") pod \"8c6d6950-7765-45f9-b233-c7351302d7a9\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " Oct 30 03:25:06.008071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.007968 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls\") pod \"8c6d6950-7765-45f9-b233-c7351302d7a9\" (UID: \"8c6d6950-7765-45f9-b233-c7351302d7a9\") " Oct 30 03:25:06.008259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.008229 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6d6950-7765-45f9-b233-c7351302d7a9-openshift-service-ca" pod "8c6d6950-7765-45f9-b233-c7351302d7a9" (UID: "8c6d6950-7765-45f9-b233-c7351302d7a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:06.008485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.008463 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6d6950-7765-45f9-b233-c7351302d7a9-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:06.010906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.010881 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6d6950-7765-45f9-b233-c7351302d7a9-kube-api-access-8s55v" pod "8c6d6950-7765-45f9-b233-c7351302d7a9" (UID: "8c6d6950-7765-45f9-b233-c7351302d7a9"). InnerVolumeSpecName "kube-api-access-8s55v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:06.011035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.011013 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls" pod "8c6d6950-7765-45f9-b233-c7351302d7a9" (UID: "8c6d6950-7765-45f9-b233-c7351302d7a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:25:06.014820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.014797 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-metrics-certs" pod "8c6d6950-7765-45f9-b233-c7351302d7a9" (UID: "8c6d6950-7765-45f9-b233-c7351302d7a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:25:06.094262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.094223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:06.120783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.109801 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:06.120783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.109867 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s55v\" (UniqueName: \"kubernetes.io/projected/8c6d6950-7765-45f9-b233-c7351302d7a9-kube-api-access-8s55v\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:06.120783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.109887 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c6d6950-7765-45f9-b233-c7351302d7a9-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:06.509473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.509442 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:06.904306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.904227 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" event={"ID":"8c6d6950-7765-45f9-b233-c7351302d7a9","Type":"ContainerDied","Data":"6b9a25332e6772a9f49a6ae84055c93cc7366a5c2c0f599e6c4e920595b5d482"} Oct 30 03:25:06.904734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.904308 4442 scope.go:122] "RemoveContainer" containerID="2139f7903649becda3d8ea26e92eed964b5bf476e2bdcb709f61ad80f1a485b6" Oct 30 03:25:06.904734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.904236 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8" Oct 30 03:25:06.953611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.953584 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8"] Oct 30 03:25:06.964316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:06.964292 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a1e5d5-566d654b6f-w7js8"] Oct 30 03:25:07.986151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:07.986115 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:08.114284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:08.114250 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:08.149677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:08.149641 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6d6950-7765-45f9-b233-c7351302d7a9" path="/var/lib/kubelet/pods/8c6d6950-7765-45f9-b233-c7351302d7a9/volumes" Oct 30 03:25:08.235269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:08.235240 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:08.719946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:08.719916 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:09.091099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:09.091063 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5"] Oct 30 03:25:09.091920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:09.091887 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" containerID="cri-o://b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0" gracePeriod=30 Oct 30 03:25:09.117197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:09.117163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:09.706938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:09.699942 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:25:09.897181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:09.897147 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d"] Oct 30 03:25:09.897363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:09.897342 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" podUID="9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" containerName="maas-api" containerID="cri-o://5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4" gracePeriod=30 Oct 30 03:25:09.970677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:09.970607 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:10.109080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.109051 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78"] Oct 30 03:25:10.109602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.109325 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" podUID="1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" containerName="payload-pre-processing" containerID="cri-o://511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f" gracePeriod=30 Oct 30 03:25:10.176627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.176594 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:10.239078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.239058 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:25:10.281140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.280284 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:10.371517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.371494 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:25:10.374670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.374647 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-maas-api-tls\") pod \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " Oct 30 03:25:10.374816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.374745 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-metrics-certs\") pod \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " Oct 30 03:25:10.374886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.374846 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-kube-api-access-l7t8d\" (UniqueName: \"kubernetes.io/projected/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-kube-api-access-l7t8d\") pod \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " Oct 30 03:25:10.374955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.374940 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-openshift-service-ca\") pod \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\" (UID: \"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad\") " Oct 30 03:25:10.375521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.375493 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-openshift-service-ca" pod "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" (UID: "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:10.376806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.376783 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-maas-api-tls" pod "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" (UID: "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:25:10.376891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.376830 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-metrics-certs" pod "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" (UID: "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:25:10.377284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.377265 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-kube-api-access-l7t8d" pod "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" (UID: "9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad"). InnerVolumeSpecName "kube-api-access-l7t8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:10.490831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.479190 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-plugins-config-volume\") pod \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " Oct 30 03:25:10.490831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.479348 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-kube-api-access-tjjrf\" (UniqueName: \"kubernetes.io/projected/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-kube-api-access-tjjrf\") pod \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\" (UID: \"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6\") " Oct 30 03:25:10.490831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.480034 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:10.490831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.480057 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:10.490831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.480071 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7t8d\" (UniqueName: \"kubernetes.io/projected/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-kube-api-access-l7t8d\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:10.490831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.480086 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:10.490831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.480479 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-plugins-config-volume" pod "1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" (UID: "1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:10.491475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.490998 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-kube-api-access-tjjrf" pod "1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" (UID: "1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6"). InnerVolumeSpecName "kube-api-access-tjjrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:10.581779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.581359 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjrf\" (UniqueName: \"kubernetes.io/projected/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-kube-api-access-tjjrf\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:10.581779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.581401 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:10.698933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.698897 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:10.825515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.825447 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:10.936045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.936007 4442 generic.go:356] "Generic (PLEG): container finished" podID="1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" containerID="511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f" exitCode=0 Oct 30 03:25:10.936221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.936063 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" Oct 30 03:25:10.936221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.936084 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" event={"ID":"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6","Type":"ContainerDied","Data":"511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f"} Oct 30 03:25:10.936221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.936175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78" event={"ID":"1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6","Type":"ContainerDied","Data":"95ab87f0d5bf18c0161930c0bef7c90d553416732539eda114dd7fb6d9f3840a"} Oct 30 03:25:10.936221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.936210 4442 scope.go:122] "RemoveContainer" containerID="511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f" Oct 30 03:25:10.937608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.937583 4442 generic.go:356] "Generic (PLEG): container finished" podID="9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" containerID="5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4" exitCode=0 Oct 30 03:25:10.937740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.937613 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" event={"ID":"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad","Type":"ContainerDied","Data":"5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4"} Oct 30 03:25:10.937740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.937639 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" Oct 30 03:25:10.937740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.937648 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d" event={"ID":"9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad","Type":"ContainerDied","Data":"3ffb27a127e9d5062b3c21b9bddbc76ccd520ba2a9a03caa241fdd0a7bdf35a6"} Oct 30 03:25:10.946637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.946621 4442 scope.go:122] "RemoveContainer" containerID="511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f" Oct 30 03:25:10.946948 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:25:10.946927 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f\": container with ID starting with 511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f not found: ID does not exist" containerID="511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f" Oct 30 03:25:10.946993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.946956 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f"} err="failed to get container status \"511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f\": rpc error: code = NotFound desc = could not find container \"511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f\": container with ID starting with 511236d7ed9d004a3c4f5983bc180dda325adb605428f87aabb4ebc265db472f not found: ID does not exist" Oct 30 03:25:10.946993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.946984 4442 scope.go:122] "RemoveContainer" containerID="5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4" Oct 30 03:25:10.955499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.955477 4442 scope.go:122] "RemoveContainer" containerID="5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4" Oct 30 03:25:10.955797 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:25:10.955777 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4\": container with ID starting with 5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4 not found: ID does not exist" containerID="5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4" Oct 30 03:25:10.955860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.955803 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4"} err="failed to get container status \"5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4\": rpc error: code = NotFound desc = could not find container \"5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4\": container with ID starting with 5786ca5697919a3daf4719d88aa62e0e3dc4fb1d376b756a16ed8eb1050f3fe4 not found: ID does not exist" Oct 30 03:25:10.983796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.983774 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d"] Oct 30 03:25:10.996281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:10.996253 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e0f003-56f59697bd-bxq9d"] Oct 30 03:25:11.026111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:11.026066 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78"] Oct 30 03:25:11.029222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:11.029198 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0f003-7dfc74fcf5-c9b78"] Oct 30 03:25:11.161075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:11.161003 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:11.466960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:11.466850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:12.108350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:12.108314 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" path="/var/lib/kubelet/pods/1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6/volumes" Oct 30 03:25:12.108765 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:12.108734 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" path="/var/lib/kubelet/pods/9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad/volumes" Oct 30 03:25:12.156866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:12.156841 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:12.318112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:12.318073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:13.346100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:13.346072 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:13.800866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:13.800830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:14.361156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.361100 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:25:14.435356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435332 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-credential-socket\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.435529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435365 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-kube-api-access-dz29w\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-kube-api-access-dz29w\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.435529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435410 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-token\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-token\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.435529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435476 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-data\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.435529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435499 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-podinfo\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.435771 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435586 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-socket\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.435771 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435625 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-credential-socket" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:14.435881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435740 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-certs\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.435881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435785 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-data" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:14.435881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435848 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istiod-ca-cert\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.436041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.435893 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-socket" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:14.436130 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436104 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-certs" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:14.436267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436132 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-envoy\") pod \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\" (UID: \"904c0e21-cf94-46fd-8cfc-e4edaf0105b3\") " Oct 30 03:25:14.436373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436303 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istiod-ca-cert" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:14.436561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436545 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.436617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436569 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.436617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436585 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.436617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436599 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.436772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.436618 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.438313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.437946 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-podinfo" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:25:14.438313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.438005 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-token" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:14.438791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.438742 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-envoy" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:14.438917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.438898 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-kube-api-access-dz29w" pod "904c0e21-cf94-46fd-8cfc-e4edaf0105b3" (UID: "904c0e21-cf94-46fd-8cfc-e4edaf0105b3"). InnerVolumeSpecName "kube-api-access-dz29w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:14.537803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.537777 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.537803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.537801 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.537997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.537812 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz29w\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-kube-api-access-dz29w\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.537997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.537822 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/904c0e21-cf94-46fd-8cfc-e4edaf0105b3-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:14.962721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.962682 4442 generic.go:356] "Generic (PLEG): container finished" podID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerID="b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0" exitCode=0 Oct 30 03:25:14.962908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.962719 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" event={"ID":"904c0e21-cf94-46fd-8cfc-e4edaf0105b3","Type":"ContainerDied","Data":"b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0"} Oct 30 03:25:14.962908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.962744 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" Oct 30 03:25:14.962908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.962872 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5" event={"ID":"904c0e21-cf94-46fd-8cfc-e4edaf0105b3","Type":"ContainerDied","Data":"b43e9829250036bd277903777f69f9499f2ce45c0d7fee0111ba9f66d0ba95b9"} Oct 30 03:25:14.963073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.962908 4442 scope.go:122] "RemoveContainer" containerID="b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0" Oct 30 03:25:14.972994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.972978 4442 scope.go:122] "RemoveContainer" containerID="2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58" Oct 30 03:25:14.981905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.981880 4442 scope.go:122] "RemoveContainer" containerID="b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0" Oct 30 03:25:14.982220 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:25:14.982199 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0\": container with ID starting with b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0 not found: ID does not exist" containerID="b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0" Oct 30 03:25:14.982306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.982236 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0"} err="failed to get container status \"b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0\": rpc error: code = NotFound desc = could not find container \"b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0\": container with ID starting with b16c47ca4fe962cdbd8e182a4fa58e805f0154509d8e80e0427c21761645bfb0 not found: ID does not exist" Oct 30 03:25:14.982306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.982255 4442 scope.go:122] "RemoveContainer" containerID="2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58" Oct 30 03:25:14.982513 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:25:14.982492 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58\": container with ID starting with 2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58 not found: ID does not exist" containerID="2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58" Oct 30 03:25:14.982564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:14.982518 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58"} err="failed to get container status \"2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58\": rpc error: code = NotFound desc = could not find container \"2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58\": container with ID starting with 2396421432d9fb1fcf437868e6d310c7132792cac271ce63c16cf15f36260b58 not found: ID does not exist" Oct 30 03:25:15.021324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.021296 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5"] Oct 30 03:25:15.032129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.032108 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a1e5d5-openshift-default-6b8bf98475-nf9g5"] Oct 30 03:25:15.063555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.063524 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:15.172933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.172903 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:15.496012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.495972 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8"] Oct 30 03:25:15.496524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496509 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" containerName="payload-pre-processing" Oct 30 03:25:15.496582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496525 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" containerName="payload-pre-processing" Oct 30 03:25:15.496582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496567 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8c6d6950-7765-45f9-b233-c7351302d7a9" containerName="maas-api" Oct 30 03:25:15.496582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496574 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6d6950-7765-45f9-b233-c7351302d7a9" containerName="maas-api" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496584 4442 cpu_manager.go:414] "containerMap: removing container" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496590 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496598 4442 cpu_manager.go:414] "containerMap: removing container" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496603 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496609 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" containerName="maas-api" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496617 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" containerName="maas-api" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496630 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1d1cd6b2-7eed-47d7-813c-96647c8949af" containerName="payload-pre-processing" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496635 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1cd6b2-7eed-47d7-813c-96647c8949af" containerName="payload-pre-processing" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496712 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6d6950-7765-45f9-b233-c7351302d7a9" containerName="maas-api" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496719 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496726 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" containerName="istio-proxy" Oct 30 03:25:15.496726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496731 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba39eee-d3b9-49d2-bb1d-ff061c53e0ad" containerName="maas-api" Oct 30 03:25:15.497283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496737 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1cd6b2-7eed-47d7-813c-96647c8949af" containerName="payload-pre-processing" Oct 30 03:25:15.497283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.496741 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c43e5a5-5519-45ea-b3ff-4cd2f8a6daf6" containerName="payload-pre-processing" Oct 30 03:25:15.501501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.501479 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.520052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.520024 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8"] Oct 30 03:25:15.527889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.527865 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4d9a1a-openshift-default-dockercfg-t5n7g\"" Oct 30 03:25:15.543415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.543389 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:15.548615 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.548591 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-envoy\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.548713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.548660 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-socket\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.549109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.549087 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vgzm\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-kube-api-access-9vgzm\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.549168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.549144 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-certs\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.549265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.549247 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aae8a4-a927-45ea-8059-ce588b6f4629-istiod-ca-cert\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.549330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.549307 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-credential-socket\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.549403 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.549381 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-token\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.549442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.549428 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-podinfo\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.549477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.549453 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-data\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.652942 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vgzm\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-kube-api-access-9vgzm\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653024 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-certs\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653087 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aae8a4-a927-45ea-8059-ce588b6f4629-istiod-ca-cert\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653160 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-credential-socket\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653200 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-token\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653242 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-podinfo\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653279 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-data\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653328 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-envoy\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.653432 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-socket\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.655776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.654210 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-credential-socket\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.661172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.661144 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-socket\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.663629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.661982 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-certs\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.663803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.662855 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aae8a4-a927-45ea-8059-ce588b6f4629-istiod-ca-cert\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.663912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.663062 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-data\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.678841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.677435 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-envoy\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.678841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.677995 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-podinfo\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.693549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.689370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-token\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.693549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.690202 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vgzm\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-kube-api-access-9vgzm\") pod \"e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.723790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.723531 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:15.820412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.820379 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:15.924852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:15.924822 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:16.090623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.090549 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:16.107682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.107647 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904c0e21-cf94-46fd-8cfc-e4edaf0105b3" path="/var/lib/kubelet/pods/904c0e21-cf94-46fd-8cfc-e4edaf0105b3/volumes" Oct 30 03:25:16.145894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.145870 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8"] Oct 30 03:25:16.146934 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:25:16.146898 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aae8a4_a927_45ea_8059_ce588b6f4629.slice/crio-79bc79212c49a3be11df0330eb460b40b2b6a8358739c1a9c223df0a647aa504 WatchSource:0}: Error finding container 79bc79212c49a3be11df0330eb460b40b2b6a8358739c1a9c223df0a647aa504: Status 404 returned error can't find the container with id 79bc79212c49a3be11df0330eb460b40b2b6a8358739c1a9c223df0a647aa504 Oct 30 03:25:16.149159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.149131 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:25:16.149213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.149191 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:25:16.149245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.149220 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:25:16.314896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.314853 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:16.974530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.974499 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:16.976981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.976954 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" event={"ID":"93aae8a4-a927-45ea-8059-ce588b6f4629","Type":"ContainerStarted","Data":"589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c"} Oct 30 03:25:16.977087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:16.977005 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" event={"ID":"93aae8a4-a927-45ea-8059-ce588b6f4629","Type":"ContainerStarted","Data":"79bc79212c49a3be11df0330eb460b40b2b6a8358739c1a9c223df0a647aa504"} Oct 30 03:25:17.041628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.041577 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" podStartSLOduration=2.041564065 podStartE2EDuration="2.041564065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:17.02964155 +0000 UTC m=+1341.031994746" watchObservedRunningTime="2026-10-30 03:25:17.041564065 +0000 UTC m=+1341.043917261" Oct 30 03:25:17.100551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.100519 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:17.259946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.259909 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:17.477032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.476999 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:17.821396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.821366 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:17.826869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.826843 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:17.891292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.891253 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:17.989179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.989110 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:17.993080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:17.991389 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:25:18.995109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:18.995073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:19.164076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:19.164045 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf"] Oct 30 03:25:19.279552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:19.279520 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:19.388889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:19.388852 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:19.753743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:19.753713 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:19.861399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:19.861356 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:20.077655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:20.077575 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:20.544453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:20.544416 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:20.721134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:20.721084 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:21.576101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.576070 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:21.622959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.622918 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq"] Oct 30 03:25:21.627189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.627165 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:21.645675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.645653 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq"] Oct 30 03:25:21.653120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.653072 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-4d9a1a\"" Oct 30 03:25:21.653277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.653145 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-4d9a1a-dockercfg-ms77z\"" Oct 30 03:25:21.694978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.694918 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:21.746794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.738963 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl"] Oct 30 03:25:21.760896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.760167 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:21.792704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.790477 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a597e7d-df54-4b47-9835-704b770ab906-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:21.792704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.790597 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22306-0738-45b6-9cf8-091156cfa933-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:21.792704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.790884 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jkwz\" (UniqueName: \"kubernetes.io/projected/bba22306-0738-45b6-9cf8-091156cfa933-kube-api-access-2jkwz\") pod \"payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:21.792704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.791010 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2zn\" (UniqueName: \"kubernetes.io/projected/9a597e7d-df54-4b47-9835-704b770ab906-kube-api-access-bt2zn\") pod \"payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:21.814221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.814185 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl"] Oct 30 03:25:21.899838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.892025 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22306-0738-45b6-9cf8-091156cfa933-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:21.899838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.892500 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jkwz\" (UniqueName: \"kubernetes.io/projected/bba22306-0738-45b6-9cf8-091156cfa933-kube-api-access-2jkwz\") pod \"payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:21.899838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.892688 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2zn\" (UniqueName: \"kubernetes.io/projected/9a597e7d-df54-4b47-9835-704b770ab906-kube-api-access-bt2zn\") pod \"payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:21.899838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.892886 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a597e7d-df54-4b47-9835-704b770ab906-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:21.899838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.893271 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22306-0738-45b6-9cf8-091156cfa933-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:21.899838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.893885 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a597e7d-df54-4b47-9835-704b770ab906-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:21.910963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.910934 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jkwz\" (UniqueName: \"kubernetes.io/projected/bba22306-0738-45b6-9cf8-091156cfa933-kube-api-access-2jkwz\") pod \"payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:21.911198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.911180 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2zn\" (UniqueName: \"kubernetes.io/projected/9a597e7d-df54-4b47-9835-704b770ab906-kube-api-access-bt2zn\") pod \"payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:21.957957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:21.957859 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:22.094954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:22.094923 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:22.271578 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:22.271556 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq"] Oct 30 03:25:22.274322 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:25:22.274296 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a597e7d_df54_4b47_9835_704b770ab906.slice/crio-09b081a983584b7d9b4eed440069b157247854d68233f04fbc5131b55b45b729 WatchSource:0}: Error finding container 09b081a983584b7d9b4eed440069b157247854d68233f04fbc5131b55b45b729: Status 404 returned error can't find the container with id 09b081a983584b7d9b4eed440069b157247854d68233f04fbc5131b55b45b729 Oct 30 03:25:22.397256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:22.397229 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl"] Oct 30 03:25:22.398358 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:25:22.398317 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba22306_0738_45b6_9cf8_091156cfa933.slice/crio-41486f2e44f07b652dc92224a3a5abcfe00951af23bbc6788d8327f9f1fcc9ca WatchSource:0}: Error finding container 41486f2e44f07b652dc92224a3a5abcfe00951af23bbc6788d8327f9f1fcc9ca: Status 404 returned error can't find the container with id 41486f2e44f07b652dc92224a3a5abcfe00951af23bbc6788d8327f9f1fcc9ca Oct 30 03:25:23.037137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.037101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" event={"ID":"9a597e7d-df54-4b47-9835-704b770ab906","Type":"ContainerStarted","Data":"d9f0802d68b08ece45576fc39363c609c0b859898bcbd332ad18c5b30636f21a"} Oct 30 03:25:23.037550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.037160 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" event={"ID":"9a597e7d-df54-4b47-9835-704b770ab906","Type":"ContainerStarted","Data":"09b081a983584b7d9b4eed440069b157247854d68233f04fbc5131b55b45b729"} Oct 30 03:25:23.037550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.037277 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:23.038946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.038923 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" event={"ID":"bba22306-0738-45b6-9cf8-091156cfa933","Type":"ContainerStarted","Data":"2c01f18338174691c24029e25a280e7c90b81645649ecae10bb04eb38b5c5eb1"} Oct 30 03:25:23.039068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.038959 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" event={"ID":"bba22306-0738-45b6-9cf8-091156cfa933","Type":"ContainerStarted","Data":"41486f2e44f07b652dc92224a3a5abcfe00951af23bbc6788d8327f9f1fcc9ca"} Oct 30 03:25:23.039068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.038987 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:23.074174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.074130 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" podStartSLOduration=2.074117414 podStartE2EDuration="2.074117414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:23.07212603 +0000 UTC m=+1347.074479229" watchObservedRunningTime="2026-10-30 03:25:23.074117414 +0000 UTC m=+1347.076470612" Oct 30 03:25:23.109152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.109097 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" podStartSLOduration=2.109079614 podStartE2EDuration="2.109079614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:23.106563901 +0000 UTC m=+1347.108917097" watchObservedRunningTime="2026-10-30 03:25:23.109079614 +0000 UTC m=+1347.111432812" Oct 30 03:25:23.488569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:23.488487 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:24.164327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.164292 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" podUID="30c17810-ebe5-41df-91e3-375e080d5e4d" containerName="payload-pre-processing" containerID="cri-o://c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d" gracePeriod=25 Oct 30 03:25:24.332115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.332085 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:24.430284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.430222 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:25:24.493373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.493335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:24.521473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.521444 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30c17810-ebe5-41df-91e3-375e080d5e4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c17810-ebe5-41df-91e3-375e080d5e4d-plugins-config-volume\") pod \"30c17810-ebe5-41df-91e3-375e080d5e4d\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " Oct 30 03:25:24.521616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.521499 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30c17810-ebe5-41df-91e3-375e080d5e4d-kube-api-access-pps4b\" (UniqueName: \"kubernetes.io/projected/30c17810-ebe5-41df-91e3-375e080d5e4d-kube-api-access-pps4b\") pod \"30c17810-ebe5-41df-91e3-375e080d5e4d\" (UID: \"30c17810-ebe5-41df-91e3-375e080d5e4d\") " Oct 30 03:25:24.521974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.521945 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c17810-ebe5-41df-91e3-375e080d5e4d-plugins-config-volume" pod "30c17810-ebe5-41df-91e3-375e080d5e4d" (UID: "30c17810-ebe5-41df-91e3-375e080d5e4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:24.523795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.523771 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c17810-ebe5-41df-91e3-375e080d5e4d-kube-api-access-pps4b" pod "30c17810-ebe5-41df-91e3-375e080d5e4d" (UID: "30c17810-ebe5-41df-91e3-375e080d5e4d"). InnerVolumeSpecName "kube-api-access-pps4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:24.623089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.623057 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pps4b\" (UniqueName: \"kubernetes.io/projected/30c17810-ebe5-41df-91e3-375e080d5e4d-kube-api-access-pps4b\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:24.623089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:24.623079 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c17810-ebe5-41df-91e3-375e080d5e4d-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:25:24.861165 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:25:24.861132 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:25:25.051315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.051282 4442 generic.go:356] "Generic (PLEG): container finished" podID="30c17810-ebe5-41df-91e3-375e080d5e4d" containerID="c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d" exitCode=0 Oct 30 03:25:25.051494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.051323 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" event={"ID":"30c17810-ebe5-41df-91e3-375e080d5e4d","Type":"ContainerDied","Data":"c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d"} Oct 30 03:25:25.051494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.051330 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" Oct 30 03:25:25.051494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.051362 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf" event={"ID":"30c17810-ebe5-41df-91e3-375e080d5e4d","Type":"ContainerDied","Data":"e6576c442f33c42b37105428463e1215e24cf1f95de2b491b3193684e623b33c"} Oct 30 03:25:25.051494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.051382 4442 scope.go:122] "RemoveContainer" containerID="c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d" Oct 30 03:25:25.061597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.061577 4442 scope.go:122] "RemoveContainer" containerID="c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d" Oct 30 03:25:25.061907 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:25:25.061882 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d\": container with ID starting with c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d not found: ID does not exist" containerID="c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d" Oct 30 03:25:25.061979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.061917 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d"} err="failed to get container status \"c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d\": rpc error: code = NotFound desc = could not find container \"c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d\": container with ID starting with c1ef8569cef3719b1b3815c7ae17297484514a794dda293c84ef3c2cc5cf066d not found: ID does not exist" Oct 30 03:25:25.100989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.100965 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf"] Oct 30 03:25:25.111003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:25.110979 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-5c9df5486c-l5jxf"] Oct 30 03:25:26.053009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.052981 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:26.107357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.107327 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30c17810-ebe5-41df-91e3-375e080d5e4d" path="/var/lib/kubelet/pods/30c17810-ebe5-41df-91e3-375e080d5e4d/volumes" Oct 30 03:25:26.170960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.170926 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:26.246943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.244053 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8"] Oct 30 03:25:26.246943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.245187 4442 cpu_manager.go:414] "containerMap: removing container" podUID="30c17810-ebe5-41df-91e3-375e080d5e4d" containerName="payload-pre-processing" Oct 30 03:25:26.246943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.245207 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30c17810-ebe5-41df-91e3-375e080d5e4d" containerName="payload-pre-processing" Oct 30 03:25:26.246943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.245491 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="30c17810-ebe5-41df-91e3-375e080d5e4d" containerName="payload-pre-processing" Oct 30 03:25:26.253474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.253451 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.261928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.261716 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8"] Oct 30 03:25:26.269648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.269620 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-7ce670-dockercfg-t8hq6\"" Oct 30 03:25:26.269867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.269669 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-7ce670\"" Oct 30 03:25:26.373912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.364432 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c030088f-33a0-4bc9-b88a-958baa829445-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.373912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.364555 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwlx\" (UniqueName: \"kubernetes.io/projected/c030088f-33a0-4bc9-b88a-958baa829445-kube-api-access-2gwlx\") pod \"payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.403645 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.399990 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l"] Oct 30 03:25:26.405192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.405163 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.435291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.432912 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l"] Oct 30 03:25:26.465823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.465790 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwlx\" (UniqueName: \"kubernetes.io/projected/c030088f-33a0-4bc9-b88a-958baa829445-kube-api-access-2gwlx\") pod \"payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.465989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.465904 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f284b\" (UniqueName: \"kubernetes.io/projected/65b1b8d7-0842-4604-823b-1b516627ab87-kube-api-access-f284b\") pod \"payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.465989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.465931 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b1b8d7-0842-4604-823b-1b516627ab87-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.466122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.466102 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c030088f-33a0-4bc9-b88a-958baa829445-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.466721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.466703 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c030088f-33a0-4bc9-b88a-958baa829445-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.484563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.484533 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwlx\" (UniqueName: \"kubernetes.io/projected/c030088f-33a0-4bc9-b88a-958baa829445-kube-api-access-2gwlx\") pod \"payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.567068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.567038 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f284b\" (UniqueName: \"kubernetes.io/projected/65b1b8d7-0842-4604-823b-1b516627ab87-kube-api-access-f284b\") pod \"payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.567068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.567074 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b1b8d7-0842-4604-823b-1b516627ab87-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.567744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.567723 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b1b8d7-0842-4604-823b-1b516627ab87-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.568776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.568725 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:26.584949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.584927 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f284b\" (UniqueName: \"kubernetes.io/projected/65b1b8d7-0842-4604-823b-1b516627ab87-kube-api-access-f284b\") pod \"payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.755246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.755216 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:26.871222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:26.871199 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8"] Oct 30 03:25:26.873823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:25:26.873743 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc030088f_33a0_4bc9_b88a_958baa829445.slice/crio-3ab187dfd15dfe7ee089913500b6d0bbeb4f0b0285e54d15257c698df37d4b9c WatchSource:0}: Error finding container 3ab187dfd15dfe7ee089913500b6d0bbeb4f0b0285e54d15257c698df37d4b9c: Status 404 returned error can't find the container with id 3ab187dfd15dfe7ee089913500b6d0bbeb4f0b0285e54d15257c698df37d4b9c Oct 30 03:25:27.066093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:27.066059 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" event={"ID":"c030088f-33a0-4bc9-b88a-958baa829445","Type":"ContainerStarted","Data":"1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271"} Oct 30 03:25:27.066555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:27.066110 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" event={"ID":"c030088f-33a0-4bc9-b88a-958baa829445","Type":"ContainerStarted","Data":"3ab187dfd15dfe7ee089913500b6d0bbeb4f0b0285e54d15257c698df37d4b9c"} Oct 30 03:25:27.066555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:27.066142 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:27.081524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:27.081497 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l"] Oct 30 03:25:27.082600 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:25:27.082577 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b1b8d7_0842_4604_823b_1b516627ab87.slice/crio-9b86930c3839a2beeb60810c55738fd9318ae83e6b885940b518ef61521e4987 WatchSource:0}: Error finding container 9b86930c3839a2beeb60810c55738fd9318ae83e6b885940b518ef61521e4987: Status 404 returned error can't find the container with id 9b86930c3839a2beeb60810c55738fd9318ae83e6b885940b518ef61521e4987 Oct 30 03:25:27.104801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:27.104736 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" podStartSLOduration=1.104722017 podStartE2EDuration="1.104722017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:27.102807571 +0000 UTC m=+1351.105160794" watchObservedRunningTime="2026-10-30 03:25:27.104722017 +0000 UTC m=+1351.107075214" Oct 30 03:25:27.916893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:27.916855 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:28.073867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:28.073824 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" event={"ID":"65b1b8d7-0842-4604-823b-1b516627ab87","Type":"ContainerStarted","Data":"5fcf6fd85c2f9af380be5e776ecc2eb5dd5fe5e07b3314d2bbf01142f08046a0"} Oct 30 03:25:28.074325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:28.073898 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" event={"ID":"65b1b8d7-0842-4604-823b-1b516627ab87","Type":"ContainerStarted","Data":"9b86930c3839a2beeb60810c55738fd9318ae83e6b885940b518ef61521e4987"} Oct 30 03:25:28.074325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:28.074121 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:28.112863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:28.112818 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" podStartSLOduration=2.112804508 podStartE2EDuration="2.112804508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:28.110415158 +0000 UTC m=+1352.112768355" watchObservedRunningTime="2026-10-30 03:25:28.112804508 +0000 UTC m=+1352.115157715" Oct 30 03:25:28.744574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:28.744546 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hgcd9_9f652e0b-e10a-47c6-971b-080c5b78b53a/payload-processing/0.log" Oct 30 03:25:28.972382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:28.972357 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hgcd9_9f652e0b-e10a-47c6-971b-080c5b78b53a/payload-processing/0.log" Oct 30 03:25:29.525419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:29.525386 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:30.195734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:30.195703 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:31.252083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:31.252053 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:33.135559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:33.135436 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:34.048337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:34.048306 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:25:34.048546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:34.048459 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:25:35.432383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:35.432329 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:36.533916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:36.533838 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:37.681861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.681821 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5"] Oct 30 03:25:37.687170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.687150 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.703109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.703087 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5"] Oct 30 03:25:37.710620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.710595 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-91a20d-serving-cert\"" Oct 30 03:25:37.763141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.763116 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:37.802122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.802093 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9ad419-a974-4609-9ec9-49c3734c3d81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.802251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.802133 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-metrics-certs\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.802316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.802303 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.802357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.802329 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77xc\" (UniqueName: \"kubernetes.io/projected/da9ad419-a974-4609-9ec9-49c3734c3d81-kube-api-access-t77xc\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.842013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.841983 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:37.903168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.903140 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.903332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.903172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77xc\" (UniqueName: \"kubernetes.io/projected/da9ad419-a974-4609-9ec9-49c3734c3d81-kube-api-access-t77xc\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.903332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.903240 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9ad419-a974-4609-9ec9-49c3734c3d81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.903332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.903266 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-metrics-certs\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.903961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.903939 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9ad419-a974-4609-9ec9-49c3734c3d81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.905826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.905798 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-metrics-certs\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.905932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.905891 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.920629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.920607 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77xc\" (UniqueName: \"kubernetes.io/projected/da9ad419-a974-4609-9ec9-49c3734c3d81-kube-api-access-t77xc\") pod \"maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:37.941821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:37.941746 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:38.012441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:38.006137 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:38.078820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:38.078000 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:25:38.318434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:38.315415 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5"] Oct 30 03:25:38.685256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:38.685177 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:25:39.083225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:39.083196 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:25:39.145092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:39.145043 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" event={"ID":"da9ad419-a974-4609-9ec9-49c3734c3d81","Type":"ContainerStarted","Data":"24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373"} Oct 30 03:25:39.145320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:39.145294 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" event={"ID":"da9ad419-a974-4609-9ec9-49c3734c3d81","Type":"ContainerStarted","Data":"7c7b167356456253252b524593457acf61396a827a8c7bca712d4c2b31043daf"} Oct 30 03:25:39.145417 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:39.145395 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:39.184398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:39.184349 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" podStartSLOduration=1.7955080319999999 podStartE2EDuration="2.184335198s" totalImagesPullingTime="388.827166ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:39.181464636 +0000 UTC m=+1363.183817835" watchObservedRunningTime="2026-10-30 03:25:39.184335198 +0000 UTC m=+1363.186688394" Oct 30 03:25:40.404670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:40.404641 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:40.776902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:40.776876 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:40.943058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:40.942975 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:41.206090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.206019 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5"] Oct 30 03:25:41.210350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.210328 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.219734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.219556 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5"] Oct 30 03:25:41.232843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.232818 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-efbbc5\"" Oct 30 03:25:41.255985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.243302 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-efbbc5-dockercfg-frmf4\"" Oct 30 03:25:41.277291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.277264 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8l2l\" (UniqueName: \"kubernetes.io/projected/1fd1f205-ced9-46d3-a53d-d678b20d95f6-kube-api-access-g8l2l\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.277494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.277473 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd1f205-ced9-46d3-a53d-d678b20d95f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.361249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.361217 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8"] Oct 30 03:25:41.365462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.365442 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.378377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.378350 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f3c7cd-9c20-40b6-af33-1cf241af4f84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.378534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.378491 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd1f205-ced9-46d3-a53d-d678b20d95f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.378619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.378553 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8l2l\" (UniqueName: \"kubernetes.io/projected/1fd1f205-ced9-46d3-a53d-d678b20d95f6-kube-api-access-g8l2l\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.378682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.378655 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scdd\" (UniqueName: \"kubernetes.io/projected/59f3c7cd-9c20-40b6-af33-1cf241af4f84-kube-api-access-2scdd\") pod \"payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.379643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.379620 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd1f205-ced9-46d3-a53d-d678b20d95f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.382339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.382318 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8"] Oct 30 03:25:41.396630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.396607 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8l2l\" (UniqueName: \"kubernetes.io/projected/1fd1f205-ced9-46d3-a53d-d678b20d95f6-kube-api-access-g8l2l\") pod \"payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.479553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.479479 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f3c7cd-9c20-40b6-af33-1cf241af4f84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.479934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.479615 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scdd\" (UniqueName: \"kubernetes.io/projected/59f3c7cd-9c20-40b6-af33-1cf241af4f84-kube-api-access-2scdd\") pod \"payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.480159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.480139 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f3c7cd-9c20-40b6-af33-1cf241af4f84-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.497093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.497060 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scdd\" (UniqueName: \"kubernetes.io/projected/59f3c7cd-9c20-40b6-af33-1cf241af4f84-kube-api-access-2scdd\") pod \"payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.537486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.537461 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:41.678043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.678008 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:41.841080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.839864 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5"] Oct 30 03:25:41.927292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.927261 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:41.997122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:41.997033 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8"] Oct 30 03:25:42.025199 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:25:42.025170 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f3c7cd_9c20_40b6_af33_1cf241af4f84.slice/crio-7821bb1799198bc12210cdb3f21182b40079ff37a263415bb9766f233425b3c4 WatchSource:0}: Error finding container 7821bb1799198bc12210cdb3f21182b40079ff37a263415bb9766f233425b3c4: Status 404 returned error can't find the container with id 7821bb1799198bc12210cdb3f21182b40079ff37a263415bb9766f233425b3c4 Oct 30 03:25:42.182687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.182652 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" event={"ID":"1fd1f205-ced9-46d3-a53d-d678b20d95f6","Type":"ContainerStarted","Data":"5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe"} Oct 30 03:25:42.182893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.182715 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" event={"ID":"1fd1f205-ced9-46d3-a53d-d678b20d95f6","Type":"ContainerStarted","Data":"bb651072eef9417e3f4fe9c9a208a9d38e8674eca45265821eb91b7f4bbc2b9d"} Oct 30 03:25:42.182893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.182744 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:42.184602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.184574 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" event={"ID":"59f3c7cd-9c20-40b6-af33-1cf241af4f84","Type":"ContainerStarted","Data":"510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02"} Oct 30 03:25:42.184721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.184624 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" event={"ID":"59f3c7cd-9c20-40b6-af33-1cf241af4f84","Type":"ContainerStarted","Data":"7821bb1799198bc12210cdb3f21182b40079ff37a263415bb9766f233425b3c4"} Oct 30 03:25:42.184721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.184652 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:42.219480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.219433 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" podStartSLOduration=1.219421015 podStartE2EDuration="1.219421015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:42.217501475 +0000 UTC m=+1366.219854675" watchObservedRunningTime="2026-10-30 03:25:42.219421015 +0000 UTC m=+1366.221774212" Oct 30 03:25:42.256238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:42.256193 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" podStartSLOduration=1.256181382 podStartE2EDuration="1.256181382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:42.25266133 +0000 UTC m=+1366.255014529" watchObservedRunningTime="2026-10-30 03:25:42.256181382 +0000 UTC m=+1366.258534579" Oct 30 03:25:43.558366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:43.558327 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:43.671293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:43.664031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:43.850125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:43.850025 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:44.818501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:44.818456 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:45.156849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:45.156770 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:25:45.390924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:45.390891 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:47.114467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:47.114424 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:47.893539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:47.893505 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:48.683568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.683528 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt"] Oct 30 03:25:48.687766 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.687730 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.703696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.703675 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-7ce670-serving-cert\"" Oct 30 03:25:48.714911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.714886 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt"] Oct 30 03:25:48.771252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.771224 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:48.811555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.811528 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.811728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.811630 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h579m\" (UniqueName: \"kubernetes.io/projected/583d5e5d-f958-4312-9d61-7b6caf149247-kube-api-access-h579m\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.811860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.811844 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-metrics-certs\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.811932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.811896 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583d5e5d-f958-4312-9d61-7b6caf149247-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.866848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.866825 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:48.913537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.913510 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-metrics-certs\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.913640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.913552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583d5e5d-f958-4312-9d61-7b6caf149247-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.913640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.913605 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.913740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.913656 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h579m\" (UniqueName: \"kubernetes.io/projected/583d5e5d-f958-4312-9d61-7b6caf149247-kube-api-access-h579m\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.914401 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.914377 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583d5e5d-f958-4312-9d61-7b6caf149247-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.916159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.916136 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.916247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.916174 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-metrics-certs\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.931695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.931674 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h579m\" (UniqueName: \"kubernetes.io/projected/583d5e5d-f958-4312-9d61-7b6caf149247-kube-api-access-h579m\") pod \"maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:48.973707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:48.973602 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:49.021776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:49.007741 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:49.376814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:49.376786 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt"] Oct 30 03:25:49.377842 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:25:49.377814 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583d5e5d_f958_4312_9d61_7b6caf149247.slice/crio-363d3117e0c5ba4121781f1c5d5a9dda747ddbb4847c6e17c1d41efd0b57d290 WatchSource:0}: Error finding container 363d3117e0c5ba4121781f1c5d5a9dda747ddbb4847c6e17c1d41efd0b57d290: Status 404 returned error can't find the container with id 363d3117e0c5ba4121781f1c5d5a9dda747ddbb4847c6e17c1d41efd0b57d290 Oct 30 03:25:50.282894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:50.282857 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" event={"ID":"583d5e5d-f958-4312-9d61-7b6caf149247","Type":"ContainerStarted","Data":"ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab"} Oct 30 03:25:50.283288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:50.282915 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" event={"ID":"583d5e5d-f958-4312-9d61-7b6caf149247","Type":"ContainerStarted","Data":"363d3117e0c5ba4121781f1c5d5a9dda747ddbb4847c6e17c1d41efd0b57d290"} Oct 30 03:25:50.283288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:50.282992 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:50.327495 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:50.327379 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" podStartSLOduration=1.993603577 podStartE2EDuration="2.327359423s" totalImagesPullingTime="333.755846ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:50.321108114 +0000 UTC m=+1374.323461312" watchObservedRunningTime="2026-10-30 03:25:50.327359423 +0000 UTC m=+1374.329712620" Oct 30 03:25:50.678881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:50.678799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:52.814517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:52.814488 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:53.192639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:53.192562 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:25:53.192797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:53.192723 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:25:54.624966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:54.624931 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:54.861158 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:25:54.861121 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:25:56.096618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:56.096581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:56.294057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:56.294032 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:25:57.716511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:57.716471 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:58.385902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:58.385865 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:59.401642 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:59.401610 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:25:59.792292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:25:59.792257 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:26:01.421234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.421203 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4"] Oct 30 03:26:01.421640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.421437 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" podUID="abaa8e42-da33-4f8d-a727-3e30e729f934" containerName="maas-api" containerID="cri-o://6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841" gracePeriod=30 Oct 30 03:26:01.485783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.481798 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:01.673462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.673365 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:01.749490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.749461 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:26:01.770686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.770652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:01.841443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.841409 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abaa8e42-da33-4f8d-a727-3e30e729f934-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaa8e42-da33-4f8d-a727-3e30e729f934-openshift-service-ca\") pod \"abaa8e42-da33-4f8d-a727-3e30e729f934\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " Oct 30 03:26:01.841639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.841518 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-metrics-certs\") pod \"abaa8e42-da33-4f8d-a727-3e30e729f934\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " Oct 30 03:26:01.841639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.841565 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls\") pod \"abaa8e42-da33-4f8d-a727-3e30e729f934\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " Oct 30 03:26:01.841795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.841664 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abaa8e42-da33-4f8d-a727-3e30e729f934-kube-api-access-c65vn\" (UniqueName: \"kubernetes.io/projected/abaa8e42-da33-4f8d-a727-3e30e729f934-kube-api-access-c65vn\") pod \"abaa8e42-da33-4f8d-a727-3e30e729f934\" (UID: \"abaa8e42-da33-4f8d-a727-3e30e729f934\") " Oct 30 03:26:01.841910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.841883 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abaa8e42-da33-4f8d-a727-3e30e729f934-openshift-service-ca" pod "abaa8e42-da33-4f8d-a727-3e30e729f934" (UID: "abaa8e42-da33-4f8d-a727-3e30e729f934"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:26:01.842200 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.842174 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abaa8e42-da33-4f8d-a727-3e30e729f934-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:01.844054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.844028 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls" pod "abaa8e42-da33-4f8d-a727-3e30e729f934" (UID: "abaa8e42-da33-4f8d-a727-3e30e729f934"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:26:01.844185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.844167 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abaa8e42-da33-4f8d-a727-3e30e729f934-kube-api-access-c65vn" pod "abaa8e42-da33-4f8d-a727-3e30e729f934" (UID: "abaa8e42-da33-4f8d-a727-3e30e729f934"). InnerVolumeSpecName "kube-api-access-c65vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:01.844287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.844262 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-metrics-certs" pod "abaa8e42-da33-4f8d-a727-3e30e729f934" (UID: "abaa8e42-da33-4f8d-a727-3e30e729f934"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:26:01.959848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.952026 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:01.959848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.952074 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abaa8e42-da33-4f8d-a727-3e30e729f934-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:01.959848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:01.952089 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65vn\" (UniqueName: \"kubernetes.io/projected/abaa8e42-da33-4f8d-a727-3e30e729f934-kube-api-access-c65vn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:02.208058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.208027 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:02.365921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.365884 4442 generic.go:356] "Generic (PLEG): container finished" podID="abaa8e42-da33-4f8d-a727-3e30e729f934" containerID="6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841" exitCode=0 Oct 30 03:26:02.366125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.365938 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" Oct 30 03:26:02.366125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.365957 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" event={"ID":"abaa8e42-da33-4f8d-a727-3e30e729f934","Type":"ContainerDied","Data":"6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841"} Oct 30 03:26:02.366125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.366011 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4" event={"ID":"abaa8e42-da33-4f8d-a727-3e30e729f934","Type":"ContainerDied","Data":"16f0bcf1559bcdcaf46c7924044bd141428a49283a661b8c20d2a53e674505a1"} Oct 30 03:26:02.366125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.366034 4442 scope.go:122] "RemoveContainer" containerID="6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841" Oct 30 03:26:02.375611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.375592 4442 scope.go:122] "RemoveContainer" containerID="6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841" Oct 30 03:26:02.375927 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:02.375909 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841\": container with ID starting with 6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841 not found: ID does not exist" containerID="6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841" Oct 30 03:26:02.375997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.375935 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841"} err="failed to get container status \"6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841\": rpc error: code = NotFound desc = could not find container \"6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841\": container with ID starting with 6c441f3d323db2512cafa84008237e931c4e1880657db6a899b09e064ff59841 not found: ID does not exist" Oct 30 03:26:02.420739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.420713 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4"] Oct 30 03:26:02.431342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:02.431313 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-499bca-84f7c56f77-lffm4"] Oct 30 03:26:04.062428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:04.062391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:04.115743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:04.115696 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abaa8e42-da33-4f8d-a727-3e30e729f934" path="/var/lib/kubelet/pods/abaa8e42-da33-4f8d-a727-3e30e729f934/volumes" Oct 30 03:26:05.940726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:05.940694 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:06.459552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:06.459518 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:07.212539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.212502 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:07.612248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.612214 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l"] Oct 30 03:26:07.612783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.612746 4442 cpu_manager.go:414] "containerMap: removing container" podUID="abaa8e42-da33-4f8d-a727-3e30e729f934" containerName="maas-api" Oct 30 03:26:07.612898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.612781 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abaa8e42-da33-4f8d-a727-3e30e729f934" containerName="maas-api" Oct 30 03:26:07.612898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.612903 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="abaa8e42-da33-4f8d-a727-3e30e729f934" containerName="maas-api" Oct 30 03:26:07.616959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.616929 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.630864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.630838 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-3eea00-openshift-default-dockercfg-ghxlr\"" Oct 30 03:26:07.645990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.645958 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l"] Oct 30 03:26:07.668569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.668531 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-certs\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.668740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.668580 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601cb276-c253-4da1-8b55-0d13035b4334-istiod-ca-cert\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.668740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.668714 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-istio-token\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.668876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.668807 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-credential-socket\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.668876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.668850 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9p9m\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-kube-api-access-x9p9m\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.668977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.668893 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601cb276-c253-4da1-8b55-0d13035b4334-istio-podinfo\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.668977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.668925 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-socket\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.669135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.669114 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-envoy\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.669201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.669182 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-data\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.677326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.677298 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770255 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-socket\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770354 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-envoy\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770406 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-data\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770471 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-certs\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770495 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601cb276-c253-4da1-8b55-0d13035b4334-istiod-ca-cert\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770538 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-istio-token\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770572 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-credential-socket\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770607 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9p9m\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-kube-api-access-x9p9m\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.771183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.770657 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601cb276-c253-4da1-8b55-0d13035b4334-istio-podinfo\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.778483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.776221 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-certs\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.778483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.776474 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-socket\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.778483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.777023 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-data\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.778483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.777976 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601cb276-c253-4da1-8b55-0d13035b4334-istiod-ca-cert\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.778483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.778213 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-credential-socket\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.779663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.779617 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601cb276-c253-4da1-8b55-0d13035b4334-istio-podinfo\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.809988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.807574 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-envoy\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.813707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.813648 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9p9m\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-kube-api-access-x9p9m\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.829905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.829820 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-istio-token\") pod \"e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:07.909420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.909323 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:07.934878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:07.934850 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:08.241252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.241112 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:08.255327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.255299 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l"] Oct 30 03:26:08.256734 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:26:08.256707 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601cb276_c253_4da1_8b55_0d13035b4334.slice/crio-d6544143a5108cfdd8b6d44570d3e3f8fb42a8727ec22201c38aa4a81d698339 WatchSource:0}: Error finding container d6544143a5108cfdd8b6d44570d3e3f8fb42a8727ec22201c38aa4a81d698339: Status 404 returned error can't find the container with id d6544143a5108cfdd8b6d44570d3e3f8fb42a8727ec22201c38aa4a81d698339 Oct 30 03:26:08.259244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.259211 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:08.259347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.259292 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:08.259347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.259334 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:08.416001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.415968 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" event={"ID":"601cb276-c253-4da1-8b55-0d13035b4334","Type":"ContainerStarted","Data":"2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6"} Oct 30 03:26:08.416188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.416018 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" event={"ID":"601cb276-c253-4da1-8b55-0d13035b4334","Type":"ContainerStarted","Data":"d6544143a5108cfdd8b6d44570d3e3f8fb42a8727ec22201c38aa4a81d698339"} Oct 30 03:26:08.465783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.465673 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podStartSLOduration=1.46565403 podStartE2EDuration="1.46565403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:08.463099247 +0000 UTC m=+1392.465452443" watchObservedRunningTime="2026-10-30 03:26:08.46565403 +0000 UTC m=+1392.468007232" Oct 30 03:26:08.934986 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:08.934955 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:09.908126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:09.908097 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:09.942984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:09.942862 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:10.008636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.008594 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:10.216862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.216741 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:10.428615 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.428580 4442 generic.go:356] "Generic (PLEG): container finished" podID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerID="589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c" exitCode=0 Oct 30 03:26:10.428840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.428652 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" event={"ID":"93aae8a4-a927-45ea-8059-ce588b6f4629","Type":"ContainerDied","Data":"589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c"} Oct 30 03:26:10.429116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.429096 4442 scope.go:122] "RemoveContainer" containerID="589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c" Oct 30 03:26:10.429202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.429157 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:10.430250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.430235 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:10.430563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.430539 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:10.430618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.430604 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:10.430655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.430642 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:10.629486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:10.629451 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:11.030568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:11.030539 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:11.121045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:11.121013 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:11.436225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:11.436139 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" event={"ID":"93aae8a4-a927-45ea-8059-ce588b6f4629","Type":"ContainerStarted","Data":"e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e"} Oct 30 03:26:11.820606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:11.820570 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:26:11.820836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:11.820815 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 03:26:11.820888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:11.820873 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 03:26:11.901591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:11.901354 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:12.222131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:12.222046 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:12.775142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:12.775101 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:12.822248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:12.822140 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 03:26:12.822248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:12.822208 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 03:26:12.872065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:12.872033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:13.821239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:13.821206 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 03:26:13.821661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:13.821261 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 03:26:14.147633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:14.147556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:14.821419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:14.821380 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 03:26:14.821951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:14.821444 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 03:26:15.728144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:15.728098 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:15.750943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:15.750914 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp"] Oct 30 03:26:15.751145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:15.751124 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" containerID="cri-o://a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d" gracePeriod=30 Oct 30 03:26:15.821280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:15.821242 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 03:26:15.821444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:15.821297 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 03:26:16.138635 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:16.138605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:16.912065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:16.912029 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:26:16.913142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:16.913120 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:26:16.913962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:16.913943 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:26:17.798260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:17.798219 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:26:17.798431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:17.798276 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:26:17.910644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:17.910609 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:18.007026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:18.006992 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:18.665252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:18.665207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:20.431057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:20.431023 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:20.531035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:20.531004 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:21.028016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.027953 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:26:21.113332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113302 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-credential-socket\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113344 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-data\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113391 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-certs\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70f81e79-58a2-40e4-aa2a-464b18728733-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f81e79-58a2-40e4-aa2a-464b18728733-istio-podinfo\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113448 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-istio-token\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-istio-token\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113468 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-socket\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113489 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-envoy\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113603 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-credential-socket" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:21.113866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113614 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-kube-api-access-vvc5g\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-kube-api-access-vvc5g\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113630 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-data" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:21.113866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113686 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-certs" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:21.113866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113713 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f81e79-58a2-40e4-aa2a-464b18728733-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f81e79-58a2-40e4-aa2a-464b18728733-istiod-ca-cert\") pod \"70f81e79-58a2-40e4-aa2a-464b18728733\" (UID: \"70f81e79-58a2-40e4-aa2a-464b18728733\") " Oct 30 03:26:21.113866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.113829 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-socket" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:21.114176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.114082 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f81e79-58a2-40e4-aa2a-464b18728733-istiod-ca-cert" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:26:21.114294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.114275 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.114353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.114299 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.114353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.114312 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.114353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.114325 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.114353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.114338 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f81e79-58a2-40e4-aa2a-464b18728733-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.115979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.115892 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-istio-token" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:21.116249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.116227 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-envoy" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:26:21.116360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.116338 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f81e79-58a2-40e4-aa2a-464b18728733-istio-podinfo" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:26:21.116488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.116416 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-kube-api-access-vvc5g" pod "70f81e79-58a2-40e4-aa2a-464b18728733" (UID: "70f81e79-58a2-40e4-aa2a-464b18728733"). InnerVolumeSpecName "kube-api-access-vvc5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:21.215779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.215717 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f81e79-58a2-40e4-aa2a-464b18728733-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.215779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.215747 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.215779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.215787 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f81e79-58a2-40e4-aa2a-464b18728733-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.216003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.215796 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvc5g\" (UniqueName: \"kubernetes.io/projected/70f81e79-58a2-40e4-aa2a-464b18728733-kube-api-access-vvc5g\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:21.521793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.521737 4442 generic.go:356] "Generic (PLEG): container finished" podID="70f81e79-58a2-40e4-aa2a-464b18728733" containerID="a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d" exitCode=0 Oct 30 03:26:21.522253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.521801 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" event={"ID":"70f81e79-58a2-40e4-aa2a-464b18728733","Type":"ContainerDied","Data":"a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d"} Oct 30 03:26:21.522253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.521849 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" event={"ID":"70f81e79-58a2-40e4-aa2a-464b18728733","Type":"ContainerDied","Data":"ebb6d7d68dee06d6b325d46eb70e9d785c930baf986903ee441fcffa40f12c17"} Oct 30 03:26:21.522253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.521879 4442 scope.go:122] "RemoveContainer" containerID="a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d" Oct 30 03:26:21.522253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.521806 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp" Oct 30 03:26:21.533009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.532991 4442 scope.go:122] "RemoveContainer" containerID="84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9" Oct 30 03:26:21.543112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.543091 4442 scope.go:122] "RemoveContainer" containerID="a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d" Oct 30 03:26:21.543419 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:21.543396 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d\": container with ID starting with a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d not found: ID does not exist" containerID="a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d" Oct 30 03:26:21.543478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.543431 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d"} err="failed to get container status \"a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d\": rpc error: code = NotFound desc = could not find container \"a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d\": container with ID starting with a21efb795e61213c04e01185ad16a9ad97d7bd8776d433bd049046fa4607469d not found: ID does not exist" Oct 30 03:26:21.543478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.543450 4442 scope.go:122] "RemoveContainer" containerID="84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9" Oct 30 03:26:21.543706 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:21.543688 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9\": container with ID starting with 84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9 not found: ID does not exist" containerID="84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9" Oct 30 03:26:21.543872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.543717 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9"} err="failed to get container status \"84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9\": rpc error: code = NotFound desc = could not find container \"84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9\": container with ID starting with 84a0547b3ddd5b533f1d66358a615adfa06d465a1a6505899484347b67e564d9 not found: ID does not exist" Oct 30 03:26:21.570479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.570450 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp"] Oct 30 03:26:21.580973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.580951 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-xxvnp"] Oct 30 03:26:21.668262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:21.668234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:22.086970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:22.086936 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:22.110689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:22.110658 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" path="/var/lib/kubelet/pods/70f81e79-58a2-40e4-aa2a-464b18728733/volumes" Oct 30 03:26:22.225787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:22.224075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:22.338917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:22.338836 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:23.724560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:23.724523 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn"] Oct 30 03:26:23.759234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:23.759195 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:23.867681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:23.867645 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:23.971509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:23.971272 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:24.714164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:24.714134 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:24.860129 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:24.860098 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:26:25.352409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.352380 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2"] Oct 30 03:26:25.352646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.352623 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" podUID="eadf6862-cb6d-449d-833e-67ccbb64cb19" containerName="maas-api" containerID="cri-o://d7438c57a6fe79c8922dff7d9ddd83cc8d3ff070eab1a9aee0d835ea3bcf64fc" gracePeriod=30 Oct 30 03:26:25.412985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.412956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:25.444603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.444570 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:26:25.557451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.557419 4442 generic.go:356] "Generic (PLEG): container finished" podID="eadf6862-cb6d-449d-833e-67ccbb64cb19" containerID="d7438c57a6fe79c8922dff7d9ddd83cc8d3ff070eab1a9aee0d835ea3bcf64fc" exitCode=0 Oct 30 03:26:25.557620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.557495 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" event={"ID":"eadf6862-cb6d-449d-833e-67ccbb64cb19","Type":"ContainerDied","Data":"d7438c57a6fe79c8922dff7d9ddd83cc8d3ff070eab1a9aee0d835ea3bcf64fc"} Oct 30 03:26:25.607277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.607203 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:25.631500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.631478 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:26:25.712186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.712152 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:25.789163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.789137 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eadf6862-cb6d-449d-833e-67ccbb64cb19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadf6862-cb6d-449d-833e-67ccbb64cb19-openshift-service-ca\") pod \"eadf6862-cb6d-449d-833e-67ccbb64cb19\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " Oct 30 03:26:25.789318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.789206 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-maas-api-tls\") pod \"eadf6862-cb6d-449d-833e-67ccbb64cb19\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " Oct 30 03:26:25.789368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.789315 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-metrics-certs\") pod \"eadf6862-cb6d-449d-833e-67ccbb64cb19\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " Oct 30 03:26:25.789469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.789432 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eadf6862-cb6d-449d-833e-67ccbb64cb19-kube-api-access-ntkvp\" (UniqueName: \"kubernetes.io/projected/eadf6862-cb6d-449d-833e-67ccbb64cb19-kube-api-access-ntkvp\") pod \"eadf6862-cb6d-449d-833e-67ccbb64cb19\" (UID: \"eadf6862-cb6d-449d-833e-67ccbb64cb19\") " Oct 30 03:26:25.789599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.789514 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eadf6862-cb6d-449d-833e-67ccbb64cb19-openshift-service-ca" pod "eadf6862-cb6d-449d-833e-67ccbb64cb19" (UID: "eadf6862-cb6d-449d-833e-67ccbb64cb19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:26:25.789823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.789806 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eadf6862-cb6d-449d-833e-67ccbb64cb19-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:25.791420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.791398 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-metrics-certs" pod "eadf6862-cb6d-449d-833e-67ccbb64cb19" (UID: "eadf6862-cb6d-449d-833e-67ccbb64cb19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:26:25.791505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.791469 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-maas-api-tls" pod "eadf6862-cb6d-449d-833e-67ccbb64cb19" (UID: "eadf6862-cb6d-449d-833e-67ccbb64cb19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:26:25.791544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.791510 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eadf6862-cb6d-449d-833e-67ccbb64cb19-kube-api-access-ntkvp" pod "eadf6862-cb6d-449d-833e-67ccbb64cb19" (UID: "eadf6862-cb6d-449d-833e-67ccbb64cb19"). InnerVolumeSpecName "kube-api-access-ntkvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:25.894830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.894690 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkvp\" (UniqueName: \"kubernetes.io/projected/eadf6862-cb6d-449d-833e-67ccbb64cb19-kube-api-access-ntkvp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:25.894830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.894736 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:25.894830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.894778 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eadf6862-cb6d-449d-833e-67ccbb64cb19-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:25.927026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:25.926200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:26.066191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.066144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:26.168022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.166523 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:26.296308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.296253 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:26.564633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.564606 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" Oct 30 03:26:26.564825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.564630 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2" event={"ID":"eadf6862-cb6d-449d-833e-67ccbb64cb19","Type":"ContainerDied","Data":"377fc49710ec5c7fafa40d43df9a125fc0107fd8ed43f29c36d204566e70ddac"} Oct 30 03:26:26.564825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.564694 4442 scope.go:122] "RemoveContainer" containerID="d7438c57a6fe79c8922dff7d9ddd83cc8d3ff070eab1a9aee0d835ea3bcf64fc" Oct 30 03:26:26.615588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.615561 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2"] Oct 30 03:26:26.626257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:26.626230 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b03a5e-8496bbbc94-wpcl2"] Oct 30 03:26:27.707829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.707797 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:27.806027 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.805998 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:27.892528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.892490 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx"] Oct 30 03:26:27.893122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893102 4442 cpu_manager.go:414] "containerMap: removing container" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" Oct 30 03:26:27.893235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893123 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" Oct 30 03:26:27.893235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893138 4442 cpu_manager.go:414] "containerMap: removing container" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" Oct 30 03:26:27.893235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893146 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" Oct 30 03:26:27.893235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893164 4442 cpu_manager.go:414] "containerMap: removing container" podUID="eadf6862-cb6d-449d-833e-67ccbb64cb19" containerName="maas-api" Oct 30 03:26:27.893235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893171 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eadf6862-cb6d-449d-833e-67ccbb64cb19" containerName="maas-api" Oct 30 03:26:27.893494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893296 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="eadf6862-cb6d-449d-833e-67ccbb64cb19" containerName="maas-api" Oct 30 03:26:27.893494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893319 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" Oct 30 03:26:27.893494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.893331 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f81e79-58a2-40e4-aa2a-464b18728733" containerName="istio-proxy" Oct 30 03:26:27.896115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.896095 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:27.925661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.925629 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx"] Oct 30 03:26:27.931240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.931215 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:27.966824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.966713 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ad5839-dockercfg-l87kp\"" Oct 30 03:26:27.978819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.978738 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs"] Oct 30 03:26:27.983228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.983208 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.003317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:27.997720 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ad5839\"" Oct 30 03:26:28.012052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.012023 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs"] Oct 30 03:26:28.039393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.038400 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cb611c-b268-419c-9691-696d6d00a23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.039393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.038646 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdmwg\" (UniqueName: \"kubernetes.io/projected/d3cb611c-b268-419c-9691-696d6d00a23b-kube-api-access-vdmwg\") pod \"payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.114389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.114356 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eadf6862-cb6d-449d-833e-67ccbb64cb19" path="/var/lib/kubelet/pods/eadf6862-cb6d-449d-833e-67ccbb64cb19/volumes" Oct 30 03:26:28.139609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.139570 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcda650-9f20-4ffe-a54a-bf06369eecde-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.139777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.139659 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrxv\" (UniqueName: \"kubernetes.io/projected/ddcda650-9f20-4ffe-a54a-bf06369eecde-kube-api-access-9zrxv\") pod \"payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.139865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.139839 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdmwg\" (UniqueName: \"kubernetes.io/projected/d3cb611c-b268-419c-9691-696d6d00a23b-kube-api-access-vdmwg\") pod \"payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.139907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.139896 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cb611c-b268-419c-9691-696d6d00a23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.140537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.140521 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cb611c-b268-419c-9691-696d6d00a23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.157763 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.157719 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdmwg\" (UniqueName: \"kubernetes.io/projected/d3cb611c-b268-419c-9691-696d6d00a23b-kube-api-access-vdmwg\") pod \"payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.208681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.208656 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.241260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.241216 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcda650-9f20-4ffe-a54a-bf06369eecde-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.241427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.241366 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrxv\" (UniqueName: \"kubernetes.io/projected/ddcda650-9f20-4ffe-a54a-bf06369eecde-kube-api-access-9zrxv\") pod \"payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.242037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.242013 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcda650-9f20-4ffe-a54a-bf06369eecde-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.256156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.256133 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:28.260519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.260019 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrxv\" (UniqueName: \"kubernetes.io/projected/ddcda650-9f20-4ffe-a54a-bf06369eecde-kube-api-access-9zrxv\") pod \"payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.341205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.334994 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:28.505005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.504964 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx"] Oct 30 03:26:28.506121 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:26:28.506097 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cb611c_b268_419c_9691_696d6d00a23b.slice/crio-bf424e5feec4123a92647fddda9ba66f2b8c57a2de55f664e0adb4a07ede7a74 WatchSource:0}: Error finding container bf424e5feec4123a92647fddda9ba66f2b8c57a2de55f664e0adb4a07ede7a74: Status 404 returned error can't find the container with id bf424e5feec4123a92647fddda9ba66f2b8c57a2de55f664e0adb4a07ede7a74 Oct 30 03:26:28.579215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.579184 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" event={"ID":"d3cb611c-b268-419c-9691-696d6d00a23b","Type":"ContainerStarted","Data":"72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc"} Oct 30 03:26:28.579333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.579247 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" event={"ID":"d3cb611c-b268-419c-9691-696d6d00a23b","Type":"ContainerStarted","Data":"bf424e5feec4123a92647fddda9ba66f2b8c57a2de55f664e0adb4a07ede7a74"} Oct 30 03:26:28.579333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.579280 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:28.615958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.615910 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" podStartSLOduration=1.615897648 podStartE2EDuration="1.615897648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:28.614259981 +0000 UTC m=+1412.616613168" watchObservedRunningTime="2026-10-30 03:26:28.615897648 +0000 UTC m=+1412.618250845" Oct 30 03:26:28.658863 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:26:28.651792 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcda650_9f20_4ffe_a54a_bf06369eecde.slice/crio-5e20b351b5927fa51a494411132af915ea4b05a57589fc819a7121a27d591ea9 WatchSource:0}: Error finding container 5e20b351b5927fa51a494411132af915ea4b05a57589fc819a7121a27d591ea9: Status 404 returned error can't find the container with id 5e20b351b5927fa51a494411132af915ea4b05a57589fc819a7121a27d591ea9 Oct 30 03:26:28.658863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.654237 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs"] Oct 30 03:26:28.725476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:28.725430 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" podUID="efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" containerName="payload-processing" containerID="cri-o://bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c" gracePeriod=25 Oct 30 03:26:29.019141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.019115 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:26:29.158497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.158468 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-kube-api-access-mss6x\" (UniqueName: \"kubernetes.io/projected/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-kube-api-access-mss6x\") pod \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " Oct 30 03:26:29.158648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.158520 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-plugins-config-volume\") pod \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\" (UID: \"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9\") " Oct 30 03:26:29.158966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.158923 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-plugins-config-volume" pod "efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" (UID: "efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:26:29.160681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.160656 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-kube-api-access-mss6x" pod "efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" (UID: "efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9"). InnerVolumeSpecName "kube-api-access-mss6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:29.260627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.260598 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mss6x\" (UniqueName: \"kubernetes.io/projected/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-kube-api-access-mss6x\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:29.260627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.260622 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:29.586175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.586138 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" event={"ID":"ddcda650-9f20-4ffe-a54a-bf06369eecde","Type":"ContainerStarted","Data":"685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a"} Oct 30 03:26:29.586368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.586193 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" event={"ID":"ddcda650-9f20-4ffe-a54a-bf06369eecde","Type":"ContainerStarted","Data":"5e20b351b5927fa51a494411132af915ea4b05a57589fc819a7121a27d591ea9"} Oct 30 03:26:29.586368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.586265 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:29.588039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.588015 4442 generic.go:356] "Generic (PLEG): container finished" podID="efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" containerID="bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c" exitCode=0 Oct 30 03:26:29.588154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.588053 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" Oct 30 03:26:29.588154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.588101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" event={"ID":"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9","Type":"ContainerDied","Data":"bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c"} Oct 30 03:26:29.588154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.588140 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn" event={"ID":"efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9","Type":"ContainerDied","Data":"bb490b929cff2389725e85409f1b1b1fcac611b34066b806643d0cffe941bf2f"} Oct 30 03:26:29.588289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.588155 4442 scope.go:122] "RemoveContainer" containerID="bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c" Oct 30 03:26:29.598197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.598183 4442 scope.go:122] "RemoveContainer" containerID="bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c" Oct 30 03:26:29.598451 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:29.598433 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c\": container with ID starting with bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c not found: ID does not exist" containerID="bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c" Oct 30 03:26:29.598504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.598457 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c"} err="failed to get container status \"bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c\": rpc error: code = NotFound desc = could not find container \"bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c\": container with ID starting with bb9362ccde6fe8c4ad843591c98740558b25047dc4e2aa0fa0667eae7fc9b17c not found: ID does not exist" Oct 30 03:26:29.624688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.624643 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" podStartSLOduration=2.624631294 podStartE2EDuration="2.624631294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:29.622252435 +0000 UTC m=+1413.624605633" watchObservedRunningTime="2026-10-30 03:26:29.624631294 +0000 UTC m=+1413.626984523" Oct 30 03:26:29.655591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.655564 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn"] Oct 30 03:26:29.658571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:29.658553 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-dff241-56c558c4bb-zq9mn"] Oct 30 03:26:30.107645 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:30.107614 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" path="/var/lib/kubelet/pods/efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9/volumes" Oct 30 03:26:30.438649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:30.438568 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:30.540394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:30.540364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:32.034634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:32.034605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:33.137192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.137162 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:33.237720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.237685 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:33.625172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.625143 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj"] Oct 30 03:26:33.625695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.625682 4442 cpu_manager.go:414] "containerMap: removing container" podUID="efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" containerName="payload-processing" Oct 30 03:26:33.625743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.625697 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" containerName="payload-processing" Oct 30 03:26:33.625807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.625802 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc4aa7e-7ba9-4e29-bf10-d47d1c5ab3f9" containerName="payload-processing" Oct 30 03:26:33.628906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.628890 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.642936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.642916 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3eea00-serving-cert\"" Oct 30 03:26:33.658096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.658064 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj"] Oct 30 03:26:33.720547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.720513 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-metrics-certs\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.720726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.720565 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c736d3ab-dd10-4872-a524-d26e151bd0f1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.720813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.720728 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbpc\" (UniqueName: \"kubernetes.io/projected/c736d3ab-dd10-4872-a524-d26e151bd0f1-kube-api-access-wzbpc\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.720813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.720794 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.783071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.783044 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:33.821786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.821741 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-metrics-certs\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.821933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.821805 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c736d3ab-dd10-4872-a524-d26e151bd0f1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.821933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.821868 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbpc\" (UniqueName: \"kubernetes.io/projected/c736d3ab-dd10-4872-a524-d26e151bd0f1-kube-api-access-wzbpc\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.821933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.821902 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.822049 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:33.822035 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-serving-cert: secret "maas-api-e2e-praxis-b-3eea00-serving-cert" not found Oct 30 03:26:33.822120 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:33.822109 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls podName:c736d3ab-dd10-4872-a524-d26e151bd0f1 nodeName:}" failed. No retries permitted until 2026-10-30 03:26:34.322092522 +0000 UTC m=+1418.324445710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls") pod "maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" (UID: "c736d3ab-dd10-4872-a524-d26e151bd0f1") : secret "maas-api-e2e-praxis-b-3eea00-serving-cert" not found Oct 30 03:26:33.822465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.822447 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c736d3ab-dd10-4872-a524-d26e151bd0f1-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.824289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.824264 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-metrics-certs\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.843058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.843033 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbpc\" (UniqueName: \"kubernetes.io/projected/c736d3ab-dd10-4872-a524-d26e151bd0f1-kube-api-access-wzbpc\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:33.877783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.877698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:33.986454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:33.986418 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:34.183127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:34.183047 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:34.327496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:34.327458 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:34.329992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:34.329964 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:34.541429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:34.541399 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:34.658240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:34.658206 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:34.854512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:34.854477 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj"] Oct 30 03:26:34.855262 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:26:34.855236 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc736d3ab_dd10_4872_a524_d26e151bd0f1.slice/crio-f28cdaeafeea2127d021867da7d1168b31ddae74aba86266699be47ebcffc756 WatchSource:0}: Error finding container f28cdaeafeea2127d021867da7d1168b31ddae74aba86266699be47ebcffc756: Status 404 returned error can't find the container with id f28cdaeafeea2127d021867da7d1168b31ddae74aba86266699be47ebcffc756 Oct 30 03:26:35.634223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:35.634176 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" event={"ID":"c736d3ab-dd10-4872-a524-d26e151bd0f1","Type":"ContainerStarted","Data":"fa67d0908a98f9c7cdb56bbede5bddb7a73a3adb4441b388583e9c7cb4a28f68"} Oct 30 03:26:35.634628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:35.634236 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" event={"ID":"c736d3ab-dd10-4872-a524-d26e151bd0f1","Type":"ContainerStarted","Data":"f28cdaeafeea2127d021867da7d1168b31ddae74aba86266699be47ebcffc756"} Oct 30 03:26:35.634628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:35.634273 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:35.672088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:35.672001 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" podStartSLOduration=2.3615664450000002 podStartE2EDuration="2.671986595s" totalImagesPullingTime="310.42015ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:35.670523995 +0000 UTC m=+1419.672877204" watchObservedRunningTime="2026-10-30 03:26:35.671986595 +0000 UTC m=+1419.674339786" Oct 30 03:26:35.708063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:35.708035 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:36.306667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:36.306632 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:36.819413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:36.819384 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:36.986347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:36.986319 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:37.516944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:37.516912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:39.176594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:39.176564 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:39.591970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:39.591938 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:26:40.075925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:40.075894 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:40.175488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:40.175463 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:40.598186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:40.598158 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:26:41.150170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:41.150136 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:41.645892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:41.645866 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:26:41.790804 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:41.790773 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:41.893177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:41.893142 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:43.622858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:43.622826 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:44.949343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:44.949309 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:45.257251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:45.257217 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:46.836008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:46.835977 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9"] Oct 30 03:26:46.836396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:46.836172 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" podUID="a59e2112-73c9-40a9-8ea6-5913e9f76c39" containerName="maas-api" containerID="cri-o://4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027" gracePeriod=30 Oct 30 03:26:46.917083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:46.917048 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:47.138830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.128738 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:26:47.219961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.219933 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59e2112-73c9-40a9-8ea6-5913e9f76c39-kube-api-access-bqnkk\" (UniqueName: \"kubernetes.io/projected/a59e2112-73c9-40a9-8ea6-5913e9f76c39-kube-api-access-bqnkk\") pod \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " Oct 30 03:26:47.220136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.219979 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls\") pod \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " Oct 30 03:26:47.220136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.220031 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59e2112-73c9-40a9-8ea6-5913e9f76c39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59e2112-73c9-40a9-8ea6-5913e9f76c39-openshift-service-ca\") pod \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " Oct 30 03:26:47.220136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.220076 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-metrics-certs\") pod \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\" (UID: \"a59e2112-73c9-40a9-8ea6-5913e9f76c39\") " Oct 30 03:26:47.220501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.220467 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59e2112-73c9-40a9-8ea6-5913e9f76c39-openshift-service-ca" pod "a59e2112-73c9-40a9-8ea6-5913e9f76c39" (UID: "a59e2112-73c9-40a9-8ea6-5913e9f76c39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:26:47.220643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.220582 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a59e2112-73c9-40a9-8ea6-5913e9f76c39-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:47.222404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.222373 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59e2112-73c9-40a9-8ea6-5913e9f76c39-kube-api-access-bqnkk" pod "a59e2112-73c9-40a9-8ea6-5913e9f76c39" (UID: "a59e2112-73c9-40a9-8ea6-5913e9f76c39"). InnerVolumeSpecName "kube-api-access-bqnkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:26:47.222404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.222385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-metrics-certs" pod "a59e2112-73c9-40a9-8ea6-5913e9f76c39" (UID: "a59e2112-73c9-40a9-8ea6-5913e9f76c39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:26:47.222538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.222442 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls" pod "a59e2112-73c9-40a9-8ea6-5913e9f76c39" (UID: "a59e2112-73c9-40a9-8ea6-5913e9f76c39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:26:47.349791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.338962 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqnkk\" (UniqueName: \"kubernetes.io/projected/a59e2112-73c9-40a9-8ea6-5913e9f76c39-kube-api-access-bqnkk\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:47.349791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.338999 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:47.349791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.339015 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a59e2112-73c9-40a9-8ea6-5913e9f76c39-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:26:47.577556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.577513 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:47.705207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.705169 4442 generic.go:356] "Generic (PLEG): container finished" podID="a59e2112-73c9-40a9-8ea6-5913e9f76c39" containerID="4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027" exitCode=0 Oct 30 03:26:47.705355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.705214 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" Oct 30 03:26:47.705355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.705256 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" event={"ID":"a59e2112-73c9-40a9-8ea6-5913e9f76c39","Type":"ContainerDied","Data":"4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027"} Oct 30 03:26:47.705355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.705310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9" event={"ID":"a59e2112-73c9-40a9-8ea6-5913e9f76c39","Type":"ContainerDied","Data":"5d2dc6b972d5a0012e83db47525b568d9ba28b5400ac24a113a0e596235af900"} Oct 30 03:26:47.705355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.705334 4442 scope.go:122] "RemoveContainer" containerID="4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027" Oct 30 03:26:47.715702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.715681 4442 scope.go:122] "RemoveContainer" containerID="4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027" Oct 30 03:26:47.715982 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:47.715962 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027\": container with ID starting with 4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027 not found: ID does not exist" containerID="4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027" Oct 30 03:26:47.716049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.715995 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027"} err="failed to get container status \"4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027\": rpc error: code = NotFound desc = could not find container \"4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027\": container with ID starting with 4a6498cf2a3f47e1b43049d469179f0e26483bf1a13485cc23b397a2662ef027 not found: ID does not exist" Oct 30 03:26:47.757087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.757055 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9"] Oct 30 03:26:47.767689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.767667 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dff241-88d5d8584-pr2g9"] Oct 30 03:26:47.888348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:47.888241 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:26:48.090416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:48.090386 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:48.107506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:48.107475 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59e2112-73c9-40a9-8ea6-5913e9f76c39" path="/var/lib/kubelet/pods/a59e2112-73c9-40a9-8ea6-5913e9f76c39/volumes" Oct 30 03:26:49.206776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.206191 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8"] Oct 30 03:26:49.207293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.207092 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a59e2112-73c9-40a9-8ea6-5913e9f76c39" containerName="maas-api" Oct 30 03:26:49.207293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.207111 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59e2112-73c9-40a9-8ea6-5913e9f76c39" containerName="maas-api" Oct 30 03:26:49.207293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.207231 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59e2112-73c9-40a9-8ea6-5913e9f76c39" containerName="maas-api" Oct 30 03:26:49.210650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.210630 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.235055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.234545 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ad5839-serving-cert\"" Oct 30 03:26:49.239519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.239482 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8"] Oct 30 03:26:49.273068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.273034 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrdvn\" (UniqueName: \"kubernetes.io/projected/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-kube-api-access-mrdvn\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.273208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.273090 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.273258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.273239 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.273523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.273501 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.307374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.307341 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:49.374218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.374178 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.374415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.374323 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.374415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.374405 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrdvn\" (UniqueName: \"kubernetes.io/projected/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-kube-api-access-mrdvn\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.374534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.374446 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.374534 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:49.374461 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-serving-cert: secret "maas-api-e2e-praxis-b-ad5839-serving-cert" not found Oct 30 03:26:49.374634 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:49.374535 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls podName:ec5752bf-35ec-4d9b-8f84-e2abe56b6650 nodeName:}" failed. No retries permitted until 2026-10-30 03:26:49.874513419 +0000 UTC m=+1433.876866598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls") pod "maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" (UID: "ec5752bf-35ec-4d9b-8f84-e2abe56b6650") : secret "maas-api-e2e-praxis-b-ad5839-serving-cert" not found Oct 30 03:26:49.375126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.375102 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.376718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.376701 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-metrics-certs\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.402693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.402626 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:49.416603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.416497 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrdvn\" (UniqueName: \"kubernetes.io/projected/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-kube-api-access-mrdvn\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.557998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.557967 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:49.881606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.881526 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:49.906777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:49.895328 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:50.123091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:50.123063 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:50.427878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:50.427847 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8"] Oct 30 03:26:50.428565 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:26:50.428542 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5752bf_35ec_4d9b_8f84_e2abe56b6650.slice/crio-ce2a4ca0c67665e14f5616595a9429f79ec8876148a72c403b97c03843be9391 WatchSource:0}: Error finding container ce2a4ca0c67665e14f5616595a9429f79ec8876148a72c403b97c03843be9391: Status 404 returned error can't find the container with id ce2a4ca0c67665e14f5616595a9429f79ec8876148a72c403b97c03843be9391 Oct 30 03:26:50.736188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:50.736107 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" event={"ID":"ec5752bf-35ec-4d9b-8f84-e2abe56b6650","Type":"ContainerStarted","Data":"ce2a4ca0c67665e14f5616595a9429f79ec8876148a72c403b97c03843be9391"} Oct 30 03:26:51.448792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:51.446079 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:51.743857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:51.743765 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" event={"ID":"ec5752bf-35ec-4d9b-8f84-e2abe56b6650","Type":"ContainerStarted","Data":"3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554"} Oct 30 03:26:51.743857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:51.743828 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:51.801779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:51.784572 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" podStartSLOduration=2.397689608 podStartE2EDuration="2.784553829s" totalImagesPullingTime="386.864221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:26:51.781326389 +0000 UTC m=+1435.783679587" watchObservedRunningTime="2026-10-30 03:26:51.784553829 +0000 UTC m=+1435.786907032" Oct 30 03:26:53.101269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:53.101231 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:54.860102 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:26:54.860056 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:26:55.609930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:55.609895 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:55.842327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:55.842297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:56.231991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:56.217807 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:56.471906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:56.471870 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:26:56.472086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:56.471928 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:26:56.697787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:56.697662 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:57.412471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:57.412419 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:57.568053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:57.568017 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:57.756462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:57.756432 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:26:58.788742 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:58.788701 4442 generic.go:356] "Generic (PLEG): container finished" podID="601cb276-c253-4da1-8b55-0d13035b4334" containerID="2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6" exitCode=0 Oct 30 03:26:58.789193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:58.788778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" event={"ID":"601cb276-c253-4da1-8b55-0d13035b4334","Type":"ContainerDied","Data":"2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6"} Oct 30 03:26:58.789193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:58.789199 4442 scope.go:122] "RemoveContainer" containerID="2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6" Oct 30 03:26:58.790840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:58.790811 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:58.790913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:58.790884 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:58.790949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:58.790920 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:26:59.293626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:59.293598 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:26:59.818576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:59.818537 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" event={"ID":"601cb276-c253-4da1-8b55-0d13035b4334","Type":"ContainerStarted","Data":"e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc"} Oct 30 03:26:59.935230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:59.935201 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:26:59.935481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:59.935414 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 03:26:59.935481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:26:59.935459 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 03:27:00.935940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:00.935903 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 03:27:00.936354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:00.935954 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 03:27:01.078587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:01.078555 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:01.911781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:01.908611 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:01.935451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:01.935417 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 03:27:01.935619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:01.935468 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 03:27:02.632189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:02.632157 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:02.660349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:02.660306 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl"] Oct 30 03:27:02.715517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:02.715473 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq"] Oct 30 03:27:02.825525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:02.825465 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:02.924005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:02.923898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:02.936267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:02.936233 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 03:27:02.937271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:02.937240 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 03:27:03.935349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:03.935313 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 03:27:03.935785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:03.935367 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 03:27:03.962857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:03.962828 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:04.710834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:04.710799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:04.806106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:04.806073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:04.906897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:04.906862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:04.935470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:04.935435 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 03:27:04.935958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:04.935491 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 03:27:05.851822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:05.851790 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:05.939891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:05.939865 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:27:05.940283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:05.940263 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:27:05.940907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:05.940889 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:27:06.537437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:06.537406 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:07.661114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:07.661054 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" podUID="bba22306-0738-45b6-9cf8-091156cfa933" containerName="payload-processing" containerID="cri-o://2c01f18338174691c24029e25a280e7c90b81645649ecae10bb04eb38b5c5eb1" gracePeriod=25 Oct 30 03:27:07.716334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:07.716287 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" podUID="9a597e7d-df54-4b47-9835-704b770ab906" containerName="payload-pre-processing" containerID="cri-o://d9f0802d68b08ece45576fc39363c609c0b859898bcbd332ad18c5b30636f21a" gracePeriod=25 Oct 30 03:27:07.904716 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:07.904682 4442 generic.go:356] "Generic (PLEG): container finished" podID="9a597e7d-df54-4b47-9835-704b770ab906" containerID="d9f0802d68b08ece45576fc39363c609c0b859898bcbd332ad18c5b30636f21a" exitCode=0 Oct 30 03:27:07.904899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:07.904825 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" event={"ID":"9a597e7d-df54-4b47-9835-704b770ab906","Type":"ContainerDied","Data":"d9f0802d68b08ece45576fc39363c609c0b859898bcbd332ad18c5b30636f21a"} Oct 30 03:27:07.906591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:07.906559 4442 generic.go:356] "Generic (PLEG): container finished" podID="bba22306-0738-45b6-9cf8-091156cfa933" containerID="2c01f18338174691c24029e25a280e7c90b81645649ecae10bb04eb38b5c5eb1" exitCode=0 Oct 30 03:27:07.907649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:07.906823 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" event={"ID":"bba22306-0738-45b6-9cf8-091156cfa933","Type":"ContainerDied","Data":"2c01f18338174691c24029e25a280e7c90b81645649ecae10bb04eb38b5c5eb1"} Oct 30 03:27:08.009841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.009812 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:27:08.041140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.041114 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:27:08.187473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.187441 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba22306-0738-45b6-9cf8-091156cfa933-kube-api-access-2jkwz\" (UniqueName: \"kubernetes.io/projected/bba22306-0738-45b6-9cf8-091156cfa933-kube-api-access-2jkwz\") pod \"bba22306-0738-45b6-9cf8-091156cfa933\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " Oct 30 03:27:08.187628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.187539 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a597e7d-df54-4b47-9835-704b770ab906-kube-api-access-bt2zn\" (UniqueName: \"kubernetes.io/projected/9a597e7d-df54-4b47-9835-704b770ab906-kube-api-access-bt2zn\") pod \"9a597e7d-df54-4b47-9835-704b770ab906\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " Oct 30 03:27:08.187628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.187577 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba22306-0738-45b6-9cf8-091156cfa933-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22306-0738-45b6-9cf8-091156cfa933-plugins-config-volume\") pod \"bba22306-0738-45b6-9cf8-091156cfa933\" (UID: \"bba22306-0738-45b6-9cf8-091156cfa933\") " Oct 30 03:27:08.187745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.187647 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a597e7d-df54-4b47-9835-704b770ab906-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a597e7d-df54-4b47-9835-704b770ab906-plugins-config-volume\") pod \"9a597e7d-df54-4b47-9835-704b770ab906\" (UID: \"9a597e7d-df54-4b47-9835-704b770ab906\") " Oct 30 03:27:08.188164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.188047 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba22306-0738-45b6-9cf8-091156cfa933-plugins-config-volume" pod "bba22306-0738-45b6-9cf8-091156cfa933" (UID: "bba22306-0738-45b6-9cf8-091156cfa933"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:08.188164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.188092 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a597e7d-df54-4b47-9835-704b770ab906-plugins-config-volume" pod "9a597e7d-df54-4b47-9835-704b770ab906" (UID: "9a597e7d-df54-4b47-9835-704b770ab906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:08.189785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.189737 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba22306-0738-45b6-9cf8-091156cfa933-kube-api-access-2jkwz" pod "bba22306-0738-45b6-9cf8-091156cfa933" (UID: "bba22306-0738-45b6-9cf8-091156cfa933"). InnerVolumeSpecName "kube-api-access-2jkwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:08.189883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.189786 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a597e7d-df54-4b47-9835-704b770ab906-kube-api-access-bt2zn" pod "9a597e7d-df54-4b47-9835-704b770ab906" (UID: "9a597e7d-df54-4b47-9835-704b770ab906"). InnerVolumeSpecName "kube-api-access-bt2zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:08.289135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.289102 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jkwz\" (UniqueName: \"kubernetes.io/projected/bba22306-0738-45b6-9cf8-091156cfa933-kube-api-access-2jkwz\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:08.289135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.289130 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt2zn\" (UniqueName: \"kubernetes.io/projected/9a597e7d-df54-4b47-9835-704b770ab906-kube-api-access-bt2zn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:08.289135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.289141 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bba22306-0738-45b6-9cf8-091156cfa933-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:08.289377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.289150 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a597e7d-df54-4b47-9835-704b770ab906-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:08.292064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.292041 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:08.914733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.914701 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" Oct 30 03:27:08.915193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.914729 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq" event={"ID":"9a597e7d-df54-4b47-9835-704b770ab906","Type":"ContainerDied","Data":"09b081a983584b7d9b4eed440069b157247854d68233f04fbc5131b55b45b729"} Oct 30 03:27:08.915193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.914830 4442 scope.go:122] "RemoveContainer" containerID="d9f0802d68b08ece45576fc39363c609c0b859898bcbd332ad18c5b30636f21a" Oct 30 03:27:08.916246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.916224 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" event={"ID":"bba22306-0738-45b6-9cf8-091156cfa933","Type":"ContainerDied","Data":"41486f2e44f07b652dc92224a3a5abcfe00951af23bbc6788d8327f9f1fcc9ca"} Oct 30 03:27:08.916312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.916239 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl" Oct 30 03:27:08.924673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.924655 4442 scope.go:122] "RemoveContainer" containerID="2c01f18338174691c24029e25a280e7c90b81645649ecae10bb04eb38b5c5eb1" Oct 30 03:27:08.961293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.961266 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl"] Oct 30 03:27:08.963594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.963574 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-4d9a1a-6d9b967d98-8jdtl"] Oct 30 03:27:08.993377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.993350 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq"] Oct 30 03:27:08.996541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:08.996520 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-4d9a1a-7d956cb475-8m2lq"] Oct 30 03:27:09.739638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:09.739609 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:10.110098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:10.110055 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a597e7d-df54-4b47-9835-704b770ab906" path="/var/lib/kubelet/pods/9a597e7d-df54-4b47-9835-704b770ab906/volumes" Oct 30 03:27:10.110711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:10.110685 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba22306-0738-45b6-9cf8-091156cfa933" path="/var/lib/kubelet/pods/bba22306-0738-45b6-9cf8-091156cfa933/volumes" Oct 30 03:27:10.405576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:10.405482 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:12.036450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:12.036411 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:12.536672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:12.536633 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:12.799773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:12.799665 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:12.932806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:12.926560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:13.929404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:13.929374 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:15.518010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:15.517970 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:16.955707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:16.955666 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5"] Oct 30 03:27:16.956238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:16.955917 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" podUID="da9ad419-a974-4609-9ec9-49c3734c3d81" containerName="maas-api" containerID="cri-o://24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373" gracePeriod=30 Oct 30 03:27:17.010022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.009993 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:17.271326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.271294 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:27:17.317182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.317151 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9ad419-a974-4609-9ec9-49c3734c3d81-kube-api-access-t77xc\" (UniqueName: \"kubernetes.io/projected/da9ad419-a974-4609-9ec9-49c3734c3d81-kube-api-access-t77xc\") pod \"da9ad419-a974-4609-9ec9-49c3734c3d81\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " Oct 30 03:27:17.317418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.317397 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-metrics-certs\") pod \"da9ad419-a974-4609-9ec9-49c3734c3d81\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " Oct 30 03:27:17.317510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.317481 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-maas-api-tls\") pod \"da9ad419-a974-4609-9ec9-49c3734c3d81\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " Oct 30 03:27:17.317570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.317540 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9ad419-a974-4609-9ec9-49c3734c3d81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9ad419-a974-4609-9ec9-49c3734c3d81-openshift-service-ca\") pod \"da9ad419-a974-4609-9ec9-49c3734c3d81\" (UID: \"da9ad419-a974-4609-9ec9-49c3734c3d81\") " Oct 30 03:27:17.321311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.321255 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-maas-api-tls" pod "da9ad419-a974-4609-9ec9-49c3734c3d81" (UID: "da9ad419-a974-4609-9ec9-49c3734c3d81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:17.321505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.321397 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-metrics-certs" pod "da9ad419-a974-4609-9ec9-49c3734c3d81" (UID: "da9ad419-a974-4609-9ec9-49c3734c3d81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:17.326882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.322437 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9ad419-a974-4609-9ec9-49c3734c3d81-openshift-service-ca" pod "da9ad419-a974-4609-9ec9-49c3734c3d81" (UID: "da9ad419-a974-4609-9ec9-49c3734c3d81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:17.326882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.323021 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:17.326882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.323045 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9ad419-a974-4609-9ec9-49c3734c3d81-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:17.326882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.323059 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9ad419-a974-4609-9ec9-49c3734c3d81-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:17.326882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.323643 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9ad419-a974-4609-9ec9-49c3734c3d81-kube-api-access-t77xc" pod "da9ad419-a974-4609-9ec9-49c3734c3d81" (UID: "da9ad419-a974-4609-9ec9-49c3734c3d81"). InnerVolumeSpecName "kube-api-access-t77xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:17.424388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.424354 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t77xc\" (UniqueName: \"kubernetes.io/projected/da9ad419-a974-4609-9ec9-49c3734c3d81-kube-api-access-t77xc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:17.789537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:17.789483 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:18.001865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.001817 4442 generic.go:356] "Generic (PLEG): container finished" podID="da9ad419-a974-4609-9ec9-49c3734c3d81" containerID="24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373" exitCode=0 Oct 30 03:27:18.001865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.001867 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" Oct 30 03:27:18.002389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.001899 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" event={"ID":"da9ad419-a974-4609-9ec9-49c3734c3d81","Type":"ContainerDied","Data":"24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373"} Oct 30 03:27:18.002389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.001979 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5" event={"ID":"da9ad419-a974-4609-9ec9-49c3734c3d81","Type":"ContainerDied","Data":"7c7b167356456253252b524593457acf61396a827a8c7bca712d4c2b31043daf"} Oct 30 03:27:18.002389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.002015 4442 scope.go:122] "RemoveContainer" containerID="24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373" Oct 30 03:27:18.014017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.014001 4442 scope.go:122] "RemoveContainer" containerID="24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373" Oct 30 03:27:18.014298 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:18.014279 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373\": container with ID starting with 24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373 not found: ID does not exist" containerID="24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373" Oct 30 03:27:18.014362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.014308 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373"} err="failed to get container status \"24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373\": rpc error: code = NotFound desc = could not find container \"24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373\": container with ID starting with 24031a3d126a21ebafd81961b90fe7df33b756339ebad86765276b53c7cdb373 not found: ID does not exist" Oct 30 03:27:18.049580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.049519 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5"] Oct 30 03:27:18.056276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.056253 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-91a20d-bfdb7db9b-wdpt5"] Oct 30 03:27:18.107445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.107414 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9ad419-a974-4609-9ec9-49c3734c3d81" path="/var/lib/kubelet/pods/da9ad419-a974-4609-9ec9-49c3734c3d81/volumes" Oct 30 03:27:18.471010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.470886 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:18.567901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:18.567862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:19.305508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:19.305477 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:19.748978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:19.748937 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:20.190836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:20.190127 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:27:20.228066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:20.228018 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:20.668511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:20.668474 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:21.769135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:21.769109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:22.098202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:22.098113 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:22.472780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:22.472661 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:24.105313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:24.105282 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:24.481383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:24.481306 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:24.581257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:24.581220 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:24.861332 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:24.861284 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:27:25.263011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:25.262960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:25.673282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:25.673198 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8"] Oct 30 03:27:25.673492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:25.673468 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" containerID="cri-o://e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e" gracePeriod=30 Oct 30 03:27:25.687881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:25.687817 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:26.096995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:26.096968 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:26.970471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:26.970447 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hgcd9_9f652e0b-e10a-47c6-971b-080c5b78b53a/payload-processing/0.log" Oct 30 03:27:27.206500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:27.206469 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-hgcd9_9f652e0b-e10a-47c6-971b-080c5b78b53a/payload-processing/0.log" Oct 30 03:27:27.392624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:27.392594 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8_59f3c7cd-9c20-40b6-af33-1cf241af4f84/payload-processing/0.log" Oct 30 03:27:27.482917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:27.482885 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:27.581099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:27.581069 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:27.597411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:27.597384 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8_59f3c7cd-9c20-40b6-af33-1cf241af4f84/payload-processing/0.log" Oct 30 03:27:27.767418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:27.767387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:27.869336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:27.869301 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:28.462539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:28.462504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:29.602978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:29.602948 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:29.995069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:29.994986 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l"] Oct 30 03:27:30.029552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.029523 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8"] Oct 30 03:27:30.128557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.128504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:30.228255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.228223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:30.725623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.725597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:30.942284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.942260 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:27:30.956874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.956815 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-credential-socket\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.956980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.956896 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93aae8a4-a927-45ea-8059-ce588b6f4629-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aae8a4-a927-45ea-8059-ce588b6f4629-istiod-ca-cert\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.956980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.956970 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-kube-api-access-9vgzm\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-kube-api-access-9vgzm\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.957089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957066 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-credential-socket" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:30.957143 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957074 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-podinfo\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.957197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957182 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-envoy\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.957275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957257 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-certs\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.957350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957279 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93aae8a4-a927-45ea-8059-ce588b6f4629-istiod-ca-cert" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:30.957350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957304 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-data\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.957350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957337 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-socket\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.957503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957371 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-token\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-token\") pod \"93aae8a4-a927-45ea-8059-ce588b6f4629\" (UID: \"93aae8a4-a927-45ea-8059-ce588b6f4629\") " Oct 30 03:27:30.957558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957520 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-certs" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:30.957613 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957593 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-data" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:30.957779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.957739 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-socket" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:30.958128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.958108 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:30.958217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.958134 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:30.958217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.958148 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:30.958217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.958161 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:30.958217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.958178 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aae8a4-a927-45ea-8059-ce588b6f4629-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:30.959485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.959457 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-kube-api-access-9vgzm" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "kube-api-access-9vgzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:30.959599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.959482 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-envoy" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:30.959599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.959545 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-podinfo" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:27:30.959978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:30.959955 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-token" pod "93aae8a4-a927-45ea-8059-ce588b6f4629" (UID: "93aae8a4-a927-45ea-8059-ce588b6f4629"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:31.051319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.051273 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:31.059874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.059850 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vgzm\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-kube-api-access-9vgzm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:31.059874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.059873 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:31.060053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.059884 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:31.060053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.059892 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aae8a4-a927-45ea-8059-ce588b6f4629-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:31.081119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.081089 4442 generic.go:356] "Generic (PLEG): container finished" podID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerID="e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e" exitCode=0 Oct 30 03:27:31.081284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.081145 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" Oct 30 03:27:31.081284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.081175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" event={"ID":"93aae8a4-a927-45ea-8059-ce588b6f4629","Type":"ContainerDied","Data":"e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e"} Oct 30 03:27:31.081284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.081227 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8" event={"ID":"93aae8a4-a927-45ea-8059-ce588b6f4629","Type":"ContainerDied","Data":"79bc79212c49a3be11df0330eb460b40b2b6a8358739c1a9c223df0a647aa504"} Oct 30 03:27:31.081284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.081252 4442 scope.go:122] "RemoveContainer" containerID="e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e" Oct 30 03:27:31.091938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.091904 4442 scope.go:122] "RemoveContainer" containerID="589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c" Oct 30 03:27:31.101123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.101106 4442 scope.go:122] "RemoveContainer" containerID="e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e" Oct 30 03:27:31.101390 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:31.101376 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e\": container with ID starting with e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e not found: ID does not exist" containerID="e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e" Oct 30 03:27:31.101435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.101398 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e"} err="failed to get container status \"e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e\": rpc error: code = NotFound desc = could not find container \"e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e\": container with ID starting with e46175143afc7a7ca500fe1557bee9eaac748e95536d8d5f8fe021be8f13a88e not found: ID does not exist" Oct 30 03:27:31.101435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.101414 4442 scope.go:122] "RemoveContainer" containerID="589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c" Oct 30 03:27:31.101665 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:31.101647 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c\": container with ID starting with 589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c not found: ID does not exist" containerID="589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c" Oct 30 03:27:31.101713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.101672 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c"} err="failed to get container status \"589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c\": rpc error: code = NotFound desc = could not find container \"589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c\": container with ID starting with 589eeabe86e0db25803d7a30a065246d2e63b2af8c0cc963139762799c1fbd9c not found: ID does not exist" Oct 30 03:27:31.148615 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.148581 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8"] Oct 30 03:27:31.152991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.152957 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4d9a1a-openshift-default-6fc54594f-slfh8"] Oct 30 03:27:31.602063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:31.602031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:32.106539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.106497 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" path="/var/lib/kubelet/pods/93aae8a4-a927-45ea-8059-ce588b6f4629/volumes" Oct 30 03:27:32.375307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.375226 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:32.819740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.819576 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm"] Oct 30 03:27:32.820215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820197 4442 cpu_manager.go:414] "containerMap: removing container" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" Oct 30 03:27:32.820215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820215 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820225 4442 cpu_manager.go:414] "containerMap: removing container" podUID="da9ad419-a974-4609-9ec9-49c3734c3d81" containerName="maas-api" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820231 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9ad419-a974-4609-9ec9-49c3734c3d81" containerName="maas-api" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820253 4442 cpu_manager.go:414] "containerMap: removing container" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820258 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820268 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9a597e7d-df54-4b47-9835-704b770ab906" containerName="payload-pre-processing" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820273 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a597e7d-df54-4b47-9835-704b770ab906" containerName="payload-pre-processing" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820288 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bba22306-0738-45b6-9cf8-091156cfa933" containerName="payload-processing" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820293 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba22306-0738-45b6-9cf8-091156cfa933" containerName="payload-processing" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820357 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820364 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a597e7d-df54-4b47-9835-704b770ab906" containerName="payload-pre-processing" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820373 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba22306-0738-45b6-9cf8-091156cfa933" containerName="payload-processing" Oct 30 03:27:32.820373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820380 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="93aae8a4-a927-45ea-8059-ce588b6f4629" containerName="istio-proxy" Oct 30 03:27:32.821009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.820385 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9ad419-a974-4609-9ec9-49c3734c3d81" containerName="maas-api" Oct 30 03:27:32.825730 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.825705 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.845637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.845528 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-dc3196-openshift-default-dockercfg-227r6\"" Oct 30 03:27:32.860349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.860324 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm"] Oct 30 03:27:32.875976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.875948 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:32.883069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883046 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-istio-token\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883074 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-envoy\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883093 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v46kc\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-kube-api-access-v46kc\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883120 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-credential-socket\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883224 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8235830-dbab-4486-8f1e-3689feb8544e-istio-podinfo\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883302 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-data\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883333 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-certs\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883479 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8235830-dbab-4486-8f1e-3689feb8544e-istiod-ca-cert\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:32.883574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.883518 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-socket\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988404 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-socket\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988475 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-istio-token\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988504 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-envoy\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988530 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v46kc\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-kube-api-access-v46kc\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988569 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-credential-socket\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988599 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8235830-dbab-4486-8f1e-3689feb8544e-istio-podinfo\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988634 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-data\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988671 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-certs\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.988856 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8235830-dbab-4486-8f1e-3689feb8544e-istiod-ca-cert\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.989658 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8235830-dbab-4486-8f1e-3689feb8544e-istiod-ca-cert\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.001781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:32.990039 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-socket\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.032975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.016501 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-credential-socket\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.032975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.019052 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-data\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.032975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.019317 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-certs\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.032975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.028311 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8235830-dbab-4486-8f1e-3689feb8544e-istio-podinfo\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.032975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.028586 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-envoy\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.045948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.045909 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-istio-token\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.046913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.046889 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v46kc\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-kube-api-access-v46kc\") pod \"e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.056600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.056571 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:33.141502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.141408 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:33.458102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.458075 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm"] Oct 30 03:27:33.459275 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:27:33.459241 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8235830_dbab_4486_8f1e_3689feb8544e.slice/crio-0219853da41c63d1c910862acf20847963c748f14f9a0ba5f139e07dccac5484 WatchSource:0}: Error finding container 0219853da41c63d1c910862acf20847963c748f14f9a0ba5f139e07dccac5484: Status 404 returned error can't find the container with id 0219853da41c63d1c910862acf20847963c748f14f9a0ba5f139e07dccac5484 Oct 30 03:27:33.462471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.462388 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:27:33.462568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.462470 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:27:33.462568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.462514 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:27:33.470567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.470548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:33.957147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:33.957117 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:34.086405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:34.086376 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:34.118722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:34.118685 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" event={"ID":"f8235830-dbab-4486-8f1e-3689feb8544e","Type":"ContainerStarted","Data":"741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99"} Oct 30 03:27:34.118898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:34.118785 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" event={"ID":"f8235830-dbab-4486-8f1e-3689feb8544e","Type":"ContainerStarted","Data":"0219853da41c63d1c910862acf20847963c748f14f9a0ba5f139e07dccac5484"} Oct 30 03:27:34.142500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:34.142474 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:34.148315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:34.148293 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:34.181541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:34.181499 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podStartSLOduration=2.181486666 podStartE2EDuration="2.181486666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:34.169610027 +0000 UTC m=+1478.171963229" watchObservedRunningTime="2026-10-30 03:27:34.181486666 +0000 UTC m=+1478.183839923" Oct 30 03:27:34.995582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:34.995546 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" podUID="65b1b8d7-0842-4604-823b-1b516627ab87" containerName="payload-processing" containerID="cri-o://5fcf6fd85c2f9af380be5e776ecc2eb5dd5fe5e07b3314d2bbf01142f08046a0" gracePeriod=25 Oct 30 03:27:35.030107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.030065 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" podUID="c030088f-33a0-4bc9-b88a-958baa829445" containerName="payload-pre-processing" containerID="cri-o://1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271" gracePeriod=25 Oct 30 03:27:35.125309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.125280 4442 generic.go:356] "Generic (PLEG): container finished" podID="65b1b8d7-0842-4604-823b-1b516627ab87" containerID="5fcf6fd85c2f9af380be5e776ecc2eb5dd5fe5e07b3314d2bbf01142f08046a0" exitCode=0 Oct 30 03:27:35.125465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.125363 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" event={"ID":"65b1b8d7-0842-4604-823b-1b516627ab87","Type":"ContainerDied","Data":"5fcf6fd85c2f9af380be5e776ecc2eb5dd5fe5e07b3314d2bbf01142f08046a0"} Oct 30 03:27:35.125629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.125615 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:35.126622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.126604 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:27:35.367113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.367091 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:27:35.368889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.368871 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:27:35.436705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.436679 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c030088f-33a0-4bc9-b88a-958baa829445-kube-api-access-2gwlx\" (UniqueName: \"kubernetes.io/projected/c030088f-33a0-4bc9-b88a-958baa829445-kube-api-access-2gwlx\") pod \"c030088f-33a0-4bc9-b88a-958baa829445\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " Oct 30 03:27:35.436705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.436717 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c030088f-33a0-4bc9-b88a-958baa829445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c030088f-33a0-4bc9-b88a-958baa829445-plugins-config-volume\") pod \"c030088f-33a0-4bc9-b88a-958baa829445\" (UID: \"c030088f-33a0-4bc9-b88a-958baa829445\") " Oct 30 03:27:35.436935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.436922 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b1b8d7-0842-4604-823b-1b516627ab87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b1b8d7-0842-4604-823b-1b516627ab87-plugins-config-volume\") pod \"65b1b8d7-0842-4604-823b-1b516627ab87\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " Oct 30 03:27:35.436998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.436988 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b1b8d7-0842-4604-823b-1b516627ab87-kube-api-access-f284b\" (UniqueName: \"kubernetes.io/projected/65b1b8d7-0842-4604-823b-1b516627ab87-kube-api-access-f284b\") pod \"65b1b8d7-0842-4604-823b-1b516627ab87\" (UID: \"65b1b8d7-0842-4604-823b-1b516627ab87\") " Oct 30 03:27:35.437262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.437227 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c030088f-33a0-4bc9-b88a-958baa829445-plugins-config-volume" pod "c030088f-33a0-4bc9-b88a-958baa829445" (UID: "c030088f-33a0-4bc9-b88a-958baa829445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:35.437358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.437307 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b1b8d7-0842-4604-823b-1b516627ab87-plugins-config-volume" pod "65b1b8d7-0842-4604-823b-1b516627ab87" (UID: "65b1b8d7-0842-4604-823b-1b516627ab87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:35.437470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.437453 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65b1b8d7-0842-4604-823b-1b516627ab87-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:35.437519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.437477 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c030088f-33a0-4bc9-b88a-958baa829445-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:35.439712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.439681 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c030088f-33a0-4bc9-b88a-958baa829445-kube-api-access-2gwlx" pod "c030088f-33a0-4bc9-b88a-958baa829445" (UID: "c030088f-33a0-4bc9-b88a-958baa829445"). InnerVolumeSpecName "kube-api-access-2gwlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:35.443528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.440278 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b1b8d7-0842-4604-823b-1b516627ab87-kube-api-access-f284b" pod "65b1b8d7-0842-4604-823b-1b516627ab87" (UID: "65b1b8d7-0842-4604-823b-1b516627ab87"). InnerVolumeSpecName "kube-api-access-f284b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:35.538464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.538384 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f284b\" (UniqueName: \"kubernetes.io/projected/65b1b8d7-0842-4604-823b-1b516627ab87-kube-api-access-f284b\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:35.538464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.538415 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gwlx\" (UniqueName: \"kubernetes.io/projected/c030088f-33a0-4bc9-b88a-958baa829445-kube-api-access-2gwlx\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:35.713582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:35.713548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:36.131821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.131786 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" event={"ID":"65b1b8d7-0842-4604-823b-1b516627ab87","Type":"ContainerDied","Data":"9b86930c3839a2beeb60810c55738fd9318ae83e6b885940b518ef61521e4987"} Oct 30 03:27:36.131821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.131807 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l" Oct 30 03:27:36.132005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.131849 4442 scope.go:122] "RemoveContainer" containerID="5fcf6fd85c2f9af380be5e776ecc2eb5dd5fe5e07b3314d2bbf01142f08046a0" Oct 30 03:27:36.133529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.133505 4442 generic.go:356] "Generic (PLEG): container finished" podID="c030088f-33a0-4bc9-b88a-958baa829445" containerID="1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271" exitCode=0 Oct 30 03:27:36.133666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.133597 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" Oct 30 03:27:36.133666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.133658 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" event={"ID":"c030088f-33a0-4bc9-b88a-958baa829445","Type":"ContainerDied","Data":"1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271"} Oct 30 03:27:36.133893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.133703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8" event={"ID":"c030088f-33a0-4bc9-b88a-958baa829445","Type":"ContainerDied","Data":"3ab187dfd15dfe7ee089913500b6d0bbeb4f0b0285e54d15257c698df37d4b9c"} Oct 30 03:27:36.141538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.141514 4442 scope.go:122] "RemoveContainer" containerID="1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271" Oct 30 03:27:36.149983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.149961 4442 scope.go:122] "RemoveContainer" containerID="1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271" Oct 30 03:27:36.150274 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:36.150252 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271\": container with ID starting with 1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271 not found: ID does not exist" containerID="1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271" Oct 30 03:27:36.150335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.150285 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271"} err="failed to get container status \"1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271\": rpc error: code = NotFound desc = could not find container \"1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271\": container with ID starting with 1afa1848d75aa84eb461eeca2dc4646d596d3c3a21c7a7c422ff283df332d271 not found: ID does not exist" Oct 30 03:27:36.181959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.181926 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l"] Oct 30 03:27:36.189557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.189531 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-7ce670-6575556d44-b6d5l"] Oct 30 03:27:36.220440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.220415 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8"] Oct 30 03:27:36.225783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.225763 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-7ce670-5c54c796c9-v5zz8"] Oct 30 03:27:36.670600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.670568 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:36.771551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:36.771520 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:37.254110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:37.254076 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:37.619850 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:37.619766 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:38.107292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.107259 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b1b8d7-0842-4604-823b-1b516627ab87" path="/var/lib/kubelet/pods/65b1b8d7-0842-4604-823b-1b516627ab87/volumes" Oct 30 03:27:38.107675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.107629 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c030088f-33a0-4bc9-b88a-958baa829445" path="/var/lib/kubelet/pods/c030088f-33a0-4bc9-b88a-958baa829445/volumes" Oct 30 03:27:38.432695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.432616 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:38.532501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.532471 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:38.614578 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.614544 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7"] Oct 30 03:27:38.615071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.615058 4442 cpu_manager.go:414] "containerMap: removing container" podUID="65b1b8d7-0842-4604-823b-1b516627ab87" containerName="payload-processing" Oct 30 03:27:38.615112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.615073 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b1b8d7-0842-4604-823b-1b516627ab87" containerName="payload-processing" Oct 30 03:27:38.615112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.615102 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c030088f-33a0-4bc9-b88a-958baa829445" containerName="payload-pre-processing" Oct 30 03:27:38.615112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.615107 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c030088f-33a0-4bc9-b88a-958baa829445" containerName="payload-pre-processing" Oct 30 03:27:38.615221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.615182 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b1b8d7-0842-4604-823b-1b516627ab87" containerName="payload-processing" Oct 30 03:27:38.615221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.615195 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c030088f-33a0-4bc9-b88a-958baa829445" containerName="payload-pre-processing" Oct 30 03:27:38.619938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.619920 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.635984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.635959 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7"] Oct 30 03:27:38.643167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.643148 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-dc3196\"" Oct 30 03:27:38.643503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.643482 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-dc3196-dockercfg-8bjnj\"" Oct 30 03:27:38.672868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.672839 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbdb317-95af-4141-8f1b-55d7eff61a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.673118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.673097 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/fbbdb317-95af-4141-8f1b-55d7eff61a27-kube-api-access-l8884\") pod \"payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.718588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.718495 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr"] Oct 30 03:27:38.744080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.744051 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:38.770274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.770240 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr"] Oct 30 03:27:38.774824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.774736 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbdb317-95af-4141-8f1b-55d7eff61a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.776021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.775992 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/fbbdb317-95af-4141-8f1b-55d7eff61a27-kube-api-access-l8884\") pod \"payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.777358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.777311 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbdb317-95af-4141-8f1b-55d7eff61a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.810024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.809995 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/fbbdb317-95af-4141-8f1b-55d7eff61a27-kube-api-access-l8884\") pod \"payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.877737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.877697 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9pvq\" (UniqueName: \"kubernetes.io/projected/dfa35cf8-883a-494f-b949-41cd36f73b41-kube-api-access-r9pvq\") pod \"payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:38.877943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.877771 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa35cf8-883a-494f-b949-41cd36f73b41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:38.933345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.933301 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:38.979644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.979562 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9pvq\" (UniqueName: \"kubernetes.io/projected/dfa35cf8-883a-494f-b949-41cd36f73b41-kube-api-access-r9pvq\") pod \"payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:38.979644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.979617 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa35cf8-883a-494f-b949-41cd36f73b41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:38.980359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:38.980338 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa35cf8-883a-494f-b949-41cd36f73b41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:39.011732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:39.011700 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9pvq\" (UniqueName: \"kubernetes.io/projected/dfa35cf8-883a-494f-b949-41cd36f73b41-kube-api-access-r9pvq\") pod \"payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:39.082325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:39.082295 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:39.232296 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:39.232270 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7"] Oct 30 03:27:39.233803 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:27:39.233739 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbdb317_95af_4141_8f1b_55d7eff61a27.slice/crio-1ec42bb5db2c25204fe69437d2d1af03444d561dabec4fa484be578d0b8e84f8 WatchSource:0}: Error finding container 1ec42bb5db2c25204fe69437d2d1af03444d561dabec4fa484be578d0b8e84f8: Status 404 returned error can't find the container with id 1ec42bb5db2c25204fe69437d2d1af03444d561dabec4fa484be578d0b8e84f8 Oct 30 03:27:39.378676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:39.378652 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr"] Oct 30 03:27:39.380013 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:27:39.379988 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa35cf8_883a_494f_b949_41cd36f73b41.slice/crio-f8a84ce2b4cc75dd2d957a7be3753bb8822f4a433ef0c1d4446868929c9ae4b4 WatchSource:0}: Error finding container f8a84ce2b4cc75dd2d957a7be3753bb8822f4a433ef0c1d4446868929c9ae4b4: Status 404 returned error can't find the container with id f8a84ce2b4cc75dd2d957a7be3753bb8822f4a433ef0c1d4446868929c9ae4b4 Oct 30 03:27:39.497996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:39.497931 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:40.165285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.165250 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" event={"ID":"fbbdb317-95af-4141-8f1b-55d7eff61a27","Type":"ContainerStarted","Data":"29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7"} Oct 30 03:27:40.165476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.165309 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" event={"ID":"fbbdb317-95af-4141-8f1b-55d7eff61a27","Type":"ContainerStarted","Data":"1ec42bb5db2c25204fe69437d2d1af03444d561dabec4fa484be578d0b8e84f8"} Oct 30 03:27:40.165476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.165419 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:40.166971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.166945 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" event={"ID":"dfa35cf8-883a-494f-b949-41cd36f73b41","Type":"ContainerStarted","Data":"12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7"} Oct 30 03:27:40.167080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.166981 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" event={"ID":"dfa35cf8-883a-494f-b949-41cd36f73b41","Type":"ContainerStarted","Data":"f8a84ce2b4cc75dd2d957a7be3753bb8822f4a433ef0c1d4446868929c9ae4b4"} Oct 30 03:27:40.167080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.167006 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:40.203522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.203478 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" podStartSLOduration=2.203465593 podStartE2EDuration="2.203465593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:40.20136591 +0000 UTC m=+1484.203719125" watchObservedRunningTime="2026-10-30 03:27:40.203465593 +0000 UTC m=+1484.205818789" Oct 30 03:27:40.238111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:40.238067 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" podStartSLOduration=2.238058372 podStartE2EDuration="2.238058372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:40.236603728 +0000 UTC m=+1484.238956927" watchObservedRunningTime="2026-10-30 03:27:40.238058372 +0000 UTC m=+1484.240411563" Oct 30 03:27:41.454214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:41.454182 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:43.559822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:43.559786 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:44.886857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:44.886823 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj"] Oct 30 03:27:44.887383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:44.887036 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" podUID="c736d3ab-dd10-4872-a524-d26e151bd0f1" containerName="maas-api" containerID="cri-o://fa67d0908a98f9c7cdb56bbede5bddb7a73a3adb4441b388583e9c7cb4a28f68" gracePeriod=30 Oct 30 03:27:44.938815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:44.938738 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:45.177145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.176985 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:45.205347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.205310 4442 generic.go:356] "Generic (PLEG): container finished" podID="c736d3ab-dd10-4872-a524-d26e151bd0f1" containerID="fa67d0908a98f9c7cdb56bbede5bddb7a73a3adb4441b388583e9c7cb4a28f68" exitCode=0 Oct 30 03:27:45.205523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.205421 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" event={"ID":"c736d3ab-dd10-4872-a524-d26e151bd0f1","Type":"ContainerDied","Data":"fa67d0908a98f9c7cdb56bbede5bddb7a73a3adb4441b388583e9c7cb4a28f68"} Oct 30 03:27:45.205523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.205467 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" event={"ID":"c736d3ab-dd10-4872-a524-d26e151bd0f1","Type":"ContainerDied","Data":"f28cdaeafeea2127d021867da7d1168b31ddae74aba86266699be47ebcffc756"} Oct 30 03:27:45.205523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.205491 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f28cdaeafeea2127d021867da7d1168b31ddae74aba86266699be47ebcffc756" Oct 30 03:27:45.205844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.205822 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:27:45.358099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.358058 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls\") pod \"c736d3ab-dd10-4872-a524-d26e151bd0f1\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " Oct 30 03:27:45.358294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.358147 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-metrics-certs\") pod \"c736d3ab-dd10-4872-a524-d26e151bd0f1\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " Oct 30 03:27:45.358294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.358190 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c736d3ab-dd10-4872-a524-d26e151bd0f1-kube-api-access-wzbpc\" (UniqueName: \"kubernetes.io/projected/c736d3ab-dd10-4872-a524-d26e151bd0f1-kube-api-access-wzbpc\") pod \"c736d3ab-dd10-4872-a524-d26e151bd0f1\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " Oct 30 03:27:45.358294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.358232 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c736d3ab-dd10-4872-a524-d26e151bd0f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c736d3ab-dd10-4872-a524-d26e151bd0f1-openshift-service-ca\") pod \"c736d3ab-dd10-4872-a524-d26e151bd0f1\" (UID: \"c736d3ab-dd10-4872-a524-d26e151bd0f1\") " Oct 30 03:27:45.358815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.358785 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c736d3ab-dd10-4872-a524-d26e151bd0f1-openshift-service-ca" pod "c736d3ab-dd10-4872-a524-d26e151bd0f1" (UID: "c736d3ab-dd10-4872-a524-d26e151bd0f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:45.400997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.389897 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls" pod "c736d3ab-dd10-4872-a524-d26e151bd0f1" (UID: "c736d3ab-dd10-4872-a524-d26e151bd0f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:45.400997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.390475 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c736d3ab-dd10-4872-a524-d26e151bd0f1-kube-api-access-wzbpc" pod "c736d3ab-dd10-4872-a524-d26e151bd0f1" (UID: "c736d3ab-dd10-4872-a524-d26e151bd0f1"). InnerVolumeSpecName "kube-api-access-wzbpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:45.400997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.398574 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-metrics-certs" pod "c736d3ab-dd10-4872-a524-d26e151bd0f1" (UID: "c736d3ab-dd10-4872-a524-d26e151bd0f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:45.459653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.459574 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:45.459653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.459605 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c736d3ab-dd10-4872-a524-d26e151bd0f1-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:45.459653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.459614 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbpc\" (UniqueName: \"kubernetes.io/projected/c736d3ab-dd10-4872-a524-d26e151bd0f1-kube-api-access-wzbpc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:45.459653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.459624 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c736d3ab-dd10-4872-a524-d26e151bd0f1-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:45.543497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:45.543458 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:46.210562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:46.210532 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj" Oct 30 03:27:46.257080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:46.257052 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj"] Oct 30 03:27:46.267850 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:46.267820 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3eea00-7dfd8d4d56-bl7rj"] Oct 30 03:27:47.128928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:47.128869 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:48.107324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:48.107292 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c736d3ab-dd10-4872-a524-d26e151bd0f1" path="/var/lib/kubelet/pods/c736d3ab-dd10-4872-a524-d26e151bd0f1/volumes" Oct 30 03:27:48.215431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:48.215397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:48.714809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:48.714775 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:49.247783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:49.247723 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:49.338997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:49.338965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:49.941605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:49.941572 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:50.191177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.191141 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt"] Oct 30 03:27:50.191341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.191320 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" podUID="583d5e5d-f958-4312-9d61-7b6caf149247" containerName="maas-api" containerID="cri-o://ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab" gracePeriod=30 Oct 30 03:27:50.248455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.248426 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:50.350350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.350239 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l"] Oct 30 03:27:50.351168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.351138 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" containerID="cri-o://e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc" gracePeriod=30 Oct 30 03:27:50.365629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.365601 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:50.557707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.557672 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:27:50.738313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.738275 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-metrics-certs\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-metrics-certs\") pod \"583d5e5d-f958-4312-9d61-7b6caf149247\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " Oct 30 03:27:50.739073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.739046 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583d5e5d-f958-4312-9d61-7b6caf149247-kube-api-access-h579m\" (UniqueName: \"kubernetes.io/projected/583d5e5d-f958-4312-9d61-7b6caf149247-kube-api-access-h579m\") pod \"583d5e5d-f958-4312-9d61-7b6caf149247\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " Oct 30 03:27:50.743945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.743913 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-maas-api-tls\") pod \"583d5e5d-f958-4312-9d61-7b6caf149247\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " Oct 30 03:27:50.744154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.744137 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583d5e5d-f958-4312-9d61-7b6caf149247-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583d5e5d-f958-4312-9d61-7b6caf149247-openshift-service-ca\") pod \"583d5e5d-f958-4312-9d61-7b6caf149247\" (UID: \"583d5e5d-f958-4312-9d61-7b6caf149247\") " Oct 30 03:27:50.767043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.766997 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583d5e5d-f958-4312-9d61-7b6caf149247-openshift-service-ca" pod "583d5e5d-f958-4312-9d61-7b6caf149247" (UID: "583d5e5d-f958-4312-9d61-7b6caf149247"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:50.789868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.780931 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-maas-api-tls" pod "583d5e5d-f958-4312-9d61-7b6caf149247" (UID: "583d5e5d-f958-4312-9d61-7b6caf149247"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:50.800792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.791163 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583d5e5d-f958-4312-9d61-7b6caf149247-kube-api-access-h579m" pod "583d5e5d-f958-4312-9d61-7b6caf149247" (UID: "583d5e5d-f958-4312-9d61-7b6caf149247"). InnerVolumeSpecName "kube-api-access-h579m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:50.800792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.797964 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-metrics-certs" pod "583d5e5d-f958-4312-9d61-7b6caf149247" (UID: "583d5e5d-f958-4312-9d61-7b6caf149247"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:27:50.846164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.846133 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:50.846164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.846164 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h579m\" (UniqueName: \"kubernetes.io/projected/583d5e5d-f958-4312-9d61-7b6caf149247-kube-api-access-h579m\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:50.846349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.846179 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/583d5e5d-f958-4312-9d61-7b6caf149247-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:50.846349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.846191 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/583d5e5d-f958-4312-9d61-7b6caf149247-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:50.942878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.942840 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:27:50.943049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:50.942912 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:27:51.111451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.111365 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:51.175600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.175551 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:27:51.175793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.175723 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:27:51.252374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.252338 4442 generic.go:356] "Generic (PLEG): container finished" podID="583d5e5d-f958-4312-9d61-7b6caf149247" containerID="ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab" exitCode=0 Oct 30 03:27:51.252374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.252376 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" Oct 30 03:27:51.252933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.252394 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" event={"ID":"583d5e5d-f958-4312-9d61-7b6caf149247","Type":"ContainerDied","Data":"ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab"} Oct 30 03:27:51.252933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.252445 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt" event={"ID":"583d5e5d-f958-4312-9d61-7b6caf149247","Type":"ContainerDied","Data":"363d3117e0c5ba4121781f1c5d5a9dda747ddbb4847c6e17c1d41efd0b57d290"} Oct 30 03:27:51.252933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.252473 4442 scope.go:122] "RemoveContainer" containerID="ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab" Oct 30 03:27:51.270724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.270705 4442 scope.go:122] "RemoveContainer" containerID="ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab" Oct 30 03:27:51.271033 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:51.271006 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab\": container with ID starting with ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab not found: ID does not exist" containerID="ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab" Oct 30 03:27:51.271119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.271040 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab"} err="failed to get container status \"ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab\": rpc error: code = NotFound desc = could not find container \"ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab\": container with ID starting with ba0bd003ec474bb62ceb841be69f8937602b630f1754e5d263b848468ea47bab not found: ID does not exist" Oct 30 03:27:51.302102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.302078 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt"] Oct 30 03:27:51.304308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:51.304286 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ce670-68fc8cd579-hr7kt"] Oct 30 03:27:52.073816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:52.073785 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:52.106270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:52.106246 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583d5e5d-f958-4312-9d61-7b6caf149247" path="/var/lib/kubelet/pods/583d5e5d-f958-4312-9d61-7b6caf149247/volumes" Oct 30 03:27:52.175658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:52.175628 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:52.749971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:52.749922 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:53.167728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:53.167650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:53.560501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:53.560466 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:54.042797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:54.042767 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:54.860079 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:54.860043 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:27:55.280485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.280453 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5"] Oct 30 03:27:55.281160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.281139 4442 cpu_manager.go:414] "containerMap: removing container" podUID="583d5e5d-f958-4312-9d61-7b6caf149247" containerName="maas-api" Oct 30 03:27:55.281253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.281163 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583d5e5d-f958-4312-9d61-7b6caf149247" containerName="maas-api" Oct 30 03:27:55.281253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.281179 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c736d3ab-dd10-4872-a524-d26e151bd0f1" containerName="maas-api" Oct 30 03:27:55.281253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.281185 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c736d3ab-dd10-4872-a524-d26e151bd0f1" containerName="maas-api" Oct 30 03:27:55.281404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.281348 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="583d5e5d-f958-4312-9d61-7b6caf149247" containerName="maas-api" Oct 30 03:27:55.281404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.281364 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c736d3ab-dd10-4872-a524-d26e151bd0f1" containerName="maas-api" Oct 30 03:27:55.286390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.286371 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.295928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.295906 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2pxn\" (UniqueName: \"kubernetes.io/projected/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-kube-api-access-t2pxn\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.296039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.296017 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.296112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.296099 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-metrics-certs\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.296169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.296127 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.312388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.312354 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-dc3196-serving-cert\"" Oct 30 03:27:55.315406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.315381 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5"] Oct 30 03:27:55.355110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.355079 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:55.367370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.367346 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:27:55.397242 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.397215 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.397338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.397292 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-metrics-certs\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.397338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.397320 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.397426 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:55.397355 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-serving-cert: secret "maas-api-e2e-praxis-b-dc3196-serving-cert" not found Oct 30 03:27:55.397426 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:55.397421 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls podName:2a1196db-cabd-47e3-a02d-4f2a64fc37a6 nodeName:}" failed. No retries permitted until 2026-10-30 03:27:55.897406705 +0000 UTC m=+1499.899759896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls") pod "maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" (UID: "2a1196db-cabd-47e3-a02d-4f2a64fc37a6") : secret "maas-api-e2e-praxis-b-dc3196-serving-cert" not found Oct 30 03:27:55.397532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.397449 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2pxn\" (UniqueName: \"kubernetes.io/projected/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-kube-api-access-t2pxn\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.398121 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.398104 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.399860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.399842 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-metrics-certs\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.417227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.417206 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2pxn\" (UniqueName: \"kubernetes.io/projected/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-kube-api-access-t2pxn\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.447058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.447021 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:55.538942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.538845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:55.680557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.680533 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:27:55.706921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.706897 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-envoy\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.706957 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-istio-token\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-istio-token\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.706985 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/601cb276-c253-4da1-8b55-0d13035b4334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601cb276-c253-4da1-8b55-0d13035b4334-istiod-ca-cert\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707005 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-certs\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707026 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-socket\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707064 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-credential-socket\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707170 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-kube-api-access-x9p9m\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-kube-api-access-x9p9m\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707233 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/601cb276-c253-4da1-8b55-0d13035b4334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601cb276-c253-4da1-8b55-0d13035b4334-istio-podinfo\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707293 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-data\") pod \"601cb276-c253-4da1-8b55-0d13035b4334\" (UID: \"601cb276-c253-4da1-8b55-0d13035b4334\") " Oct 30 03:27:55.707423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707321 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-certs" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:55.707423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707330 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-socket" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:55.707569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707433 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601cb276-c253-4da1-8b55-0d13035b4334-istiod-ca-cert" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:27:55.707811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707628 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-data" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:55.707811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707788 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.707811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707808 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/601cb276-c253-4da1-8b55-0d13035b4334-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.707978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707820 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.707978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707834 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.707978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.707781 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-credential-socket" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:55.709303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.709276 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-envoy" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:55.709411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.709318 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-istio-token" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:55.709411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.709376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/601cb276-c253-4da1-8b55-0d13035b4334-istio-podinfo" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:27:55.709564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.709547 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-kube-api-access-x9p9m" pod "601cb276-c253-4da1-8b55-0d13035b4334" (UID: "601cb276-c253-4da1-8b55-0d13035b4334"). InnerVolumeSpecName "kube-api-access-x9p9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:55.809218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.809149 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.809218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.809175 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.809218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.809184 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/601cb276-c253-4da1-8b55-0d13035b4334-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.809218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.809195 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9p9m\" (UniqueName: \"kubernetes.io/projected/601cb276-c253-4da1-8b55-0d13035b4334-kube-api-access-x9p9m\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.809218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.809204 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/601cb276-c253-4da1-8b55-0d13035b4334-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:27:55.910399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.910365 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:55.913055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:55.913033 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls\") pod \"maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:56.199568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.199537 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:56.326726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.325620 4442 generic.go:356] "Generic (PLEG): container finished" podID="601cb276-c253-4da1-8b55-0d13035b4334" containerID="e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc" exitCode=0 Oct 30 03:27:56.326726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.325734 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" event={"ID":"601cb276-c253-4da1-8b55-0d13035b4334","Type":"ContainerDied","Data":"e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc"} Oct 30 03:27:56.326726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.325861 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" event={"ID":"601cb276-c253-4da1-8b55-0d13035b4334","Type":"ContainerDied","Data":"d6544143a5108cfdd8b6d44570d3e3f8fb42a8727ec22201c38aa4a81d698339"} Oct 30 03:27:56.326726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.325889 4442 scope.go:122] "RemoveContainer" containerID="e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc" Oct 30 03:27:56.326726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.325989 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l" Oct 30 03:27:56.338383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.337139 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:27:56.338984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.338963 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:27:56.341330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.341315 4442 scope.go:122] "RemoveContainer" containerID="2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6" Oct 30 03:27:56.354997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.354976 4442 scope.go:122] "RemoveContainer" containerID="e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc" Oct 30 03:27:56.355309 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:56.355280 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc\": container with ID starting with e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc not found: ID does not exist" containerID="e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc" Oct 30 03:27:56.355390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.355324 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc"} err="failed to get container status \"e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc\": rpc error: code = NotFound desc = could not find container \"e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc\": container with ID starting with e6a53ae1b8c85b6e8aecaf093fc5e3845f6a7961f7cce0b7679be0337141badc not found: ID does not exist" Oct 30 03:27:56.355390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.355350 4442 scope.go:122] "RemoveContainer" containerID="2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6" Oct 30 03:27:56.355627 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:27:56.355605 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6\": container with ID starting with 2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6 not found: ID does not exist" containerID="2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6" Oct 30 03:27:56.355704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.355635 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6"} err="failed to get container status \"2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6\": rpc error: code = NotFound desc = could not find container \"2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6\": container with ID starting with 2f9b8e51f3a225862bbdbfc3e1f7a6d3b4d1024b74c60dd352803de6a44701e6 not found: ID does not exist" Oct 30 03:27:56.376265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.376239 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l"] Oct 30 03:27:56.389065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.389044 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-3eea00-openshift-default-bc8f4888f-sm95l"] Oct 30 03:27:56.592253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:56.592221 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5"] Oct 30 03:27:57.220904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:57.220870 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:57.332497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:57.332399 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" event={"ID":"2a1196db-cabd-47e3-a02d-4f2a64fc37a6","Type":"ContainerStarted","Data":"e3a52c7398346a49f6953fd1bbe6ac9a89f5f4340b2cff45fa08cba2a3ffea31"} Oct 30 03:27:57.332497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:57.332481 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" event={"ID":"2a1196db-cabd-47e3-a02d-4f2a64fc37a6","Type":"ContainerStarted","Data":"337e2fef2f66e68896b750b88d906fd19a106e17dd3bdd12280a1562ab525070"} Oct 30 03:27:57.333026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:57.332504 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:27:57.371687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:57.371634 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" podStartSLOduration=1.908060534 podStartE2EDuration="2.371621885s" totalImagesPullingTime="463.561351ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:27:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:57.368965736 +0000 UTC m=+1501.371318933" watchObservedRunningTime="2026-10-30 03:27:57.371621885 +0000 UTC m=+1501.373975123" Oct 30 03:27:58.108610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:58.108574 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601cb276-c253-4da1-8b55-0d13035b4334" path="/var/lib/kubelet/pods/601cb276-c253-4da1-8b55-0d13035b4334/volumes" Oct 30 03:27:58.921190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:58.921158 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:59.718107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:59.718073 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs"] Oct 30 03:27:59.756264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:59.756230 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx"] Oct 30 03:27:59.859223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:59.859181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:27:59.989663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:27:59.989554 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:00.394786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.393907 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8"] Oct 30 03:28:00.394786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.394666 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" podUID="ec5752bf-35ec-4d9b-8f84-e2abe56b6650" containerName="maas-api" containerID="cri-o://3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554" gracePeriod=30 Oct 30 03:28:00.463381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.463331 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:00.736812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.736791 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:28:00.810808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.804220 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-openshift-service-ca\") pod \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " Oct 30 03:28:00.810808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.804455 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-metrics-certs\") pod \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " Oct 30 03:28:00.810808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.804492 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls\") pod \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " Oct 30 03:28:00.810808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.804582 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-kube-api-access-mrdvn\" (UniqueName: \"kubernetes.io/projected/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-kube-api-access-mrdvn\") pod \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\" (UID: \"ec5752bf-35ec-4d9b-8f84-e2abe56b6650\") " Oct 30 03:28:00.828928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.828546 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-openshift-service-ca" pod "ec5752bf-35ec-4d9b-8f84-e2abe56b6650" (UID: "ec5752bf-35ec-4d9b-8f84-e2abe56b6650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:00.832100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.831231 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-kube-api-access-mrdvn" pod "ec5752bf-35ec-4d9b-8f84-e2abe56b6650" (UID: "ec5752bf-35ec-4d9b-8f84-e2abe56b6650"). InnerVolumeSpecName "kube-api-access-mrdvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:00.876025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.875955 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-metrics-certs" pod "ec5752bf-35ec-4d9b-8f84-e2abe56b6650" (UID: "ec5752bf-35ec-4d9b-8f84-e2abe56b6650"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:00.880606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.880545 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls" pod "ec5752bf-35ec-4d9b-8f84-e2abe56b6650" (UID: "ec5752bf-35ec-4d9b-8f84-e2abe56b6650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:00.907007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.906974 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:00.907007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.907004 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:00.907007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.907014 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrdvn\" (UniqueName: \"kubernetes.io/projected/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-kube-api-access-mrdvn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:00.907325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:00.907023 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec5752bf-35ec-4d9b-8f84-e2abe56b6650-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:01.186591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.186556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:01.371995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.371966 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:01.395089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.395061 4442 generic.go:356] "Generic (PLEG): container finished" podID="ec5752bf-35ec-4d9b-8f84-e2abe56b6650" containerID="3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554" exitCode=0 Oct 30 03:28:01.395263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.395117 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" Oct 30 03:28:01.395263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.395141 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" event={"ID":"ec5752bf-35ec-4d9b-8f84-e2abe56b6650","Type":"ContainerDied","Data":"3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554"} Oct 30 03:28:01.395263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.395190 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8" event={"ID":"ec5752bf-35ec-4d9b-8f84-e2abe56b6650","Type":"ContainerDied","Data":"ce2a4ca0c67665e14f5616595a9429f79ec8876148a72c403b97c03843be9391"} Oct 30 03:28:01.395263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.395211 4442 scope.go:122] "RemoveContainer" containerID="3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554" Oct 30 03:28:01.407915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.407894 4442 scope.go:122] "RemoveContainer" containerID="3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554" Oct 30 03:28:01.408409 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:01.408390 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554\": container with ID starting with 3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554 not found: ID does not exist" containerID="3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554" Oct 30 03:28:01.408498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.408420 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554"} err="failed to get container status \"3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554\": rpc error: code = NotFound desc = could not find container \"3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554\": container with ID starting with 3f82c91143b8c175d33dd7fc69c3427b76429f3daa4a6d0a0b355ee49625d554 not found: ID does not exist" Oct 30 03:28:01.458530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.458451 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8"] Oct 30 03:28:01.469955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:01.469930 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ad5839-d5d9cf6bb-9jcd8"] Oct 30 03:28:02.107364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:02.107322 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec5752bf-35ec-4d9b-8f84-e2abe56b6650" path="/var/lib/kubelet/pods/ec5752bf-35ec-4d9b-8f84-e2abe56b6650/volumes" Oct 30 03:28:03.344309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:03.344276 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:28:03.515809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:03.515778 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:04.718423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:04.718375 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" podUID="ddcda650-9f20-4ffe-a54a-bf06369eecde" containerName="payload-processing" containerID="cri-o://685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a" gracePeriod=25 Oct 30 03:28:04.756587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:04.756557 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" podUID="d3cb611c-b268-419c-9691-696d6d00a23b" containerName="payload-pre-processing" containerID="cri-o://72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc" gracePeriod=25 Oct 30 03:28:05.028120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.028091 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:28:05.060712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.060683 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddcda650-9f20-4ffe-a54a-bf06369eecde-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcda650-9f20-4ffe-a54a-bf06369eecde-plugins-config-volume\") pod \"ddcda650-9f20-4ffe-a54a-bf06369eecde\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " Oct 30 03:28:05.060861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.060737 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddcda650-9f20-4ffe-a54a-bf06369eecde-kube-api-access-9zrxv\" (UniqueName: \"kubernetes.io/projected/ddcda650-9f20-4ffe-a54a-bf06369eecde-kube-api-access-9zrxv\") pod \"ddcda650-9f20-4ffe-a54a-bf06369eecde\" (UID: \"ddcda650-9f20-4ffe-a54a-bf06369eecde\") " Oct 30 03:28:05.061040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.061020 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcda650-9f20-4ffe-a54a-bf06369eecde-plugins-config-volume" pod "ddcda650-9f20-4ffe-a54a-bf06369eecde" (UID: "ddcda650-9f20-4ffe-a54a-bf06369eecde"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:05.063083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.063057 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddcda650-9f20-4ffe-a54a-bf06369eecde-kube-api-access-9zrxv" pod "ddcda650-9f20-4ffe-a54a-bf06369eecde" (UID: "ddcda650-9f20-4ffe-a54a-bf06369eecde"). InnerVolumeSpecName "kube-api-access-9zrxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:05.077041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.077012 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:28:05.162061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.162029 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3cb611c-b268-419c-9691-696d6d00a23b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cb611c-b268-419c-9691-696d6d00a23b-plugins-config-volume\") pod \"d3cb611c-b268-419c-9691-696d6d00a23b\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " Oct 30 03:28:05.162246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.162090 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3cb611c-b268-419c-9691-696d6d00a23b-kube-api-access-vdmwg\" (UniqueName: \"kubernetes.io/projected/d3cb611c-b268-419c-9691-696d6d00a23b-kube-api-access-vdmwg\") pod \"d3cb611c-b268-419c-9691-696d6d00a23b\" (UID: \"d3cb611c-b268-419c-9691-696d6d00a23b\") " Oct 30 03:28:05.162334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.162318 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddcda650-9f20-4ffe-a54a-bf06369eecde-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:05.162334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.162331 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrxv\" (UniqueName: \"kubernetes.io/projected/ddcda650-9f20-4ffe-a54a-bf06369eecde-kube-api-access-9zrxv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:05.162456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.162360 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cb611c-b268-419c-9691-696d6d00a23b-plugins-config-volume" pod "d3cb611c-b268-419c-9691-696d6d00a23b" (UID: "d3cb611c-b268-419c-9691-696d6d00a23b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:05.164349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.164321 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cb611c-b268-419c-9691-696d6d00a23b-kube-api-access-vdmwg" pod "d3cb611c-b268-419c-9691-696d6d00a23b" (UID: "d3cb611c-b268-419c-9691-696d6d00a23b"). InnerVolumeSpecName "kube-api-access-vdmwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:05.263581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.263547 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3cb611c-b268-419c-9691-696d6d00a23b-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:05.263581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.263577 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdmwg\" (UniqueName: \"kubernetes.io/projected/d3cb611c-b268-419c-9691-696d6d00a23b-kube-api-access-vdmwg\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:05.316893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.316841 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:05.419530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.419499 4442 generic.go:356] "Generic (PLEG): container finished" podID="ddcda650-9f20-4ffe-a54a-bf06369eecde" containerID="685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a" exitCode=0 Oct 30 03:28:05.419688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.419549 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" Oct 30 03:28:05.419688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.419576 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" event={"ID":"ddcda650-9f20-4ffe-a54a-bf06369eecde","Type":"ContainerDied","Data":"685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a"} Oct 30 03:28:05.419688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.419629 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs" event={"ID":"ddcda650-9f20-4ffe-a54a-bf06369eecde","Type":"ContainerDied","Data":"5e20b351b5927fa51a494411132af915ea4b05a57589fc819a7121a27d591ea9"} Oct 30 03:28:05.419688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.419655 4442 scope.go:122] "RemoveContainer" containerID="685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a" Oct 30 03:28:05.421366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.421337 4442 generic.go:356] "Generic (PLEG): container finished" podID="d3cb611c-b268-419c-9691-696d6d00a23b" containerID="72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc" exitCode=0 Oct 30 03:28:05.421468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.421387 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" Oct 30 03:28:05.421468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.421423 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" event={"ID":"d3cb611c-b268-419c-9691-696d6d00a23b","Type":"ContainerDied","Data":"72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc"} Oct 30 03:28:05.421468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.421461 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx" event={"ID":"d3cb611c-b268-419c-9691-696d6d00a23b","Type":"ContainerDied","Data":"bf424e5feec4123a92647fddda9ba66f2b8c57a2de55f664e0adb4a07ede7a74"} Oct 30 03:28:05.430502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.430482 4442 scope.go:122] "RemoveContainer" containerID="685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a" Oct 30 03:28:05.430955 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:05.430817 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a\": container with ID starting with 685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a not found: ID does not exist" containerID="685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a" Oct 30 03:28:05.430955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.430850 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a"} err="failed to get container status \"685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a\": rpc error: code = NotFound desc = could not find container \"685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a\": container with ID starting with 685b37e1b863df8258ff7b0d2577dd139283f3ec9e772b01d7107aa7cd3f190a not found: ID does not exist" Oct 30 03:28:05.430955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.430871 4442 scope.go:122] "RemoveContainer" containerID="72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc" Oct 30 03:28:05.440157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.440136 4442 scope.go:122] "RemoveContainer" containerID="72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc" Oct 30 03:28:05.440440 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:05.440422 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc\": container with ID starting with 72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc not found: ID does not exist" containerID="72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc" Oct 30 03:28:05.440484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.440449 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc"} err="failed to get container status \"72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc\": rpc error: code = NotFound desc = could not find container \"72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc\": container with ID starting with 72bf7d8ea4e4587d34f9755126bbcd50b38224adb200a6ad0c37f2a2a48467dc not found: ID does not exist" Oct 30 03:28:05.466899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.466841 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx"] Oct 30 03:28:05.470570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.470545 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad5839-8578776659-dmsxx"] Oct 30 03:28:05.501668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.501633 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs"] Oct 30 03:28:05.507703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:05.507678 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ad5839-6cd595b74b-6p2rs"] Oct 30 03:28:06.106947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:06.106915 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3cb611c-b268-419c-9691-696d6d00a23b" path="/var/lib/kubelet/pods/d3cb611c-b268-419c-9691-696d6d00a23b/volumes" Oct 30 03:28:06.107314 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:06.107301 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddcda650-9f20-4ffe-a54a-bf06369eecde" path="/var/lib/kubelet/pods/ddcda650-9f20-4ffe-a54a-bf06369eecde/volumes" Oct 30 03:28:06.871702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:06.871672 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:07.002779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:07.000356 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:08.637563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:08.637531 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:10.388730 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:10.388697 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:10.946222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:10.946189 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:12.020543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.020513 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:12.810052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810017 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk"] Oct 30 03:28:12.810553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810537 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d3cb611c-b268-419c-9691-696d6d00a23b" containerName="payload-pre-processing" Oct 30 03:28:12.810625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810554 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3cb611c-b268-419c-9691-696d6d00a23b" containerName="payload-pre-processing" Oct 30 03:28:12.810625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810570 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ddcda650-9f20-4ffe-a54a-bf06369eecde" containerName="payload-processing" Oct 30 03:28:12.810625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810575 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddcda650-9f20-4ffe-a54a-bf06369eecde" containerName="payload-processing" Oct 30 03:28:12.810625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810587 4442 cpu_manager.go:414] "containerMap: removing container" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" Oct 30 03:28:12.810625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810591 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" Oct 30 03:28:12.810809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810627 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ec5752bf-35ec-4d9b-8f84-e2abe56b6650" containerName="maas-api" Oct 30 03:28:12.810809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810633 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec5752bf-35ec-4d9b-8f84-e2abe56b6650" containerName="maas-api" Oct 30 03:28:12.810809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810719 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" Oct 30 03:28:12.810809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810727 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec5752bf-35ec-4d9b-8f84-e2abe56b6650" containerName="maas-api" Oct 30 03:28:12.810809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810734 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3cb611c-b268-419c-9691-696d6d00a23b" containerName="payload-pre-processing" Oct 30 03:28:12.810809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810740 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddcda650-9f20-4ffe-a54a-bf06369eecde" containerName="payload-processing" Oct 30 03:28:12.810809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.810747 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" Oct 30 03:28:12.815719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.815700 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.836068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.836030 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:12.836245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.836160 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk"] Oct 30 03:28:12.989968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.989934 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.989988 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.990082 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.990233 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.990286 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.990369 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.990413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.990447 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7kkr\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-kube-api-access-v7kkr\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:12.990539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:12.990523 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.091877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.091786 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.091877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.091863 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.091888 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.091913 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.091943 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7kkr\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-kube-api-access-v7kkr\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.091984 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.092049 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.092099 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.092135 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.092515 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.092537 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.092638 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.092900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.092741 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.093153 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.093126 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.094335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.094313 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.094643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.094623 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.109697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.109674 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.109877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.109860 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7kkr\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-kube-api-access-v7kkr\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.129686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.129668 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:13.275897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.275866 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:28:13.457103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.457073 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk"] Oct 30 03:28:13.458492 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:28:13.458468 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf7f8a_0670_475c_9f9b_be6addc4cb5f.slice/crio-8af51baf553a0272ad6e3d9611a38861b3e6c65075695da2aaf4be7a36cefc89 WatchSource:0}: Error finding container 8af51baf553a0272ad6e3d9611a38861b3e6c65075695da2aaf4be7a36cefc89: Status 404 returned error can't find the container with id 8af51baf553a0272ad6e3d9611a38861b3e6c65075695da2aaf4be7a36cefc89 Oct 30 03:28:13.461253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.461220 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:13.461335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.461285 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:13.461335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.461317 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:13.490813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:13.490785 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" event={"ID":"0baf7f8a-0670-475c-9f9b-be6addc4cb5f","Type":"ContainerStarted","Data":"8af51baf553a0272ad6e3d9611a38861b3e6c65075695da2aaf4be7a36cefc89"} Oct 30 03:28:14.553776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:14.550036 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" event={"ID":"0baf7f8a-0670-475c-9f9b-be6addc4cb5f","Type":"ContainerStarted","Data":"6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4"} Oct 30 03:28:15.032792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:15.030166 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:15.132153 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:15.132118 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:15.139935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:15.139902 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 03:28:15.140207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:15.140175 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 03:28:16.140250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:16.140190 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 03:28:16.140909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:16.140269 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 03:28:17.126781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:17.124644 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:17.133775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:17.132192 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 03:28:17.133775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:17.132276 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 03:28:18.132779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:18.132112 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 03:28:18.132779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:18.132169 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 03:28:18.573993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:18.573961 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:19.132988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:19.132940 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 03:28:19.133525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:19.133039 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 03:28:19.337781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:19.337482 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:20.130831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:20.130728 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 03:28:20.130831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:20.130814 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 03:28:21.167983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:21.167939 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:28:21.168536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:21.168029 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:28:21.534705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:21.534647 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:21.914005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:21.913912 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:21.914484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:21.914456 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:21.915943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:21.915905 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:28:21.967913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:21.967856 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podStartSLOduration=9.967836489 podStartE2EDuration="9.967836489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:14.662563318 +0000 UTC m=+1518.664916554" watchObservedRunningTime="2026-10-30 03:28:21.967836489 +0000 UTC m=+1525.970189688" Oct 30 03:28:22.493892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:22.493862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:22.699120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:22.699086 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:22.804615 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:22.800137 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:23.364834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.364801 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:23.662320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.662235 4442 generic.go:356] "Generic (PLEG): container finished" podID="f8235830-dbab-4486-8f1e-3689feb8544e" containerID="741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99" exitCode=0 Oct 30 03:28:23.662320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.662304 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" event={"ID":"f8235830-dbab-4486-8f1e-3689feb8544e","Type":"ContainerDied","Data":"741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99"} Oct 30 03:28:23.662802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.662788 4442 scope.go:122] "RemoveContainer" containerID="741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99" Oct 30 03:28:23.664084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.664057 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:23.664197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.664138 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:23.664197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.664184 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:23.798675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.798642 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:23.967865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:23.959464 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:24.670419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:24.670390 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" event={"ID":"f8235830-dbab-4486-8f1e-3689feb8544e","Type":"ContainerStarted","Data":"07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575"} Oct 30 03:28:24.679109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:24.679074 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:24.861128 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:24.861094 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:28:25.142167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:25.142135 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:28:25.142371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:25.142321 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 03:28:25.142471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:25.142367 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 03:28:25.821958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:25.821921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:26.142355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:26.142269 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 03:28:26.142355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:26.142332 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 03:28:27.142107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:27.142077 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 03:28:27.142510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:27.142127 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 03:28:27.612793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:27.612746 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:27.874006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:27.873917 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:28.142914 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:28.142825 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 03:28:28.143496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:28.143461 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 03:28:29.142118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:29.142090 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 03:28:29.142297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:29.142155 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 03:28:29.369687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:29.369654 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:30.146322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:30.146292 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:28:30.146675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:30.146653 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:28:30.147403 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:30.147386 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:28:30.781144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:30.781112 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst"] Oct 30 03:28:30.781529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:30.781307 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" podUID="0ac811d5-6d49-4db7-a098-343417e96c71" containerName="maas-api" containerID="cri-o://1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3" gracePeriod=30 Oct 30 03:28:30.842698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:30.842663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:31.018111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.018039 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:31.062016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.061940 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:28:31.088065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.088033 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ac811d5-6d49-4db7-a098-343417e96c71-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac811d5-6d49-4db7-a098-343417e96c71-openshift-service-ca\") pod \"0ac811d5-6d49-4db7-a098-343417e96c71\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " Oct 30 03:28:31.088227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.088072 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ac811d5-6d49-4db7-a098-343417e96c71-kube-api-access-chcqx\" (UniqueName: \"kubernetes.io/projected/0ac811d5-6d49-4db7-a098-343417e96c71-kube-api-access-chcqx\") pod \"0ac811d5-6d49-4db7-a098-343417e96c71\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " Oct 30 03:28:31.088227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.088105 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-metrics-certs\") pod \"0ac811d5-6d49-4db7-a098-343417e96c71\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " Oct 30 03:28:31.088227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.088214 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-maas-api-tls\") pod \"0ac811d5-6d49-4db7-a098-343417e96c71\" (UID: \"0ac811d5-6d49-4db7-a098-343417e96c71\") " Oct 30 03:28:31.088416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.088335 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ac811d5-6d49-4db7-a098-343417e96c71-openshift-service-ca" pod "0ac811d5-6d49-4db7-a098-343417e96c71" (UID: "0ac811d5-6d49-4db7-a098-343417e96c71"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:31.088505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.088480 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ac811d5-6d49-4db7-a098-343417e96c71-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:31.090324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.090292 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-metrics-certs" pod "0ac811d5-6d49-4db7-a098-343417e96c71" (UID: "0ac811d5-6d49-4db7-a098-343417e96c71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:31.090425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.090375 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-maas-api-tls" pod "0ac811d5-6d49-4db7-a098-343417e96c71" (UID: "0ac811d5-6d49-4db7-a098-343417e96c71"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:31.090425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.090409 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ac811d5-6d49-4db7-a098-343417e96c71-kube-api-access-chcqx" pod "0ac811d5-6d49-4db7-a098-343417e96c71" (UID: "0ac811d5-6d49-4db7-a098-343417e96c71"). InnerVolumeSpecName "kube-api-access-chcqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:31.111316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.111286 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:31.140712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.140684 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn"] Oct 30 03:28:31.141460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.141442 4442 cpu_manager.go:414] "containerMap: removing container" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" Oct 30 03:28:31.141520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.141465 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601cb276-c253-4da1-8b55-0d13035b4334" containerName="istio-proxy" Oct 30 03:28:31.141520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.141508 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0ac811d5-6d49-4db7-a098-343417e96c71" containerName="maas-api" Oct 30 03:28:31.141520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.141515 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ac811d5-6d49-4db7-a098-343417e96c71" containerName="maas-api" Oct 30 03:28:31.141654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.141641 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ac811d5-6d49-4db7-a098-343417e96c71" containerName="maas-api" Oct 30 03:28:31.145388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.145371 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.160414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.160389 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8d80b1-openshift-default-dockercfg-22948\"" Oct 30 03:28:31.163128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.163107 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn"] Oct 30 03:28:31.190021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.189991 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190039 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abd507-0528-408f-adc7-cdc1fa787d54-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190167 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190228 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190261 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcmlj\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-kube-api-access-lcmlj\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190327 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190351 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190372 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190399 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abd507-0528-408f-adc7-cdc1fa787d54-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.190598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190550 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:31.190598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190572 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ac811d5-6d49-4db7-a098-343417e96c71-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:31.190598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.190589 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcqx\" (UniqueName: \"kubernetes.io/projected/0ac811d5-6d49-4db7-a098-343417e96c71-kube-api-access-chcqx\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:31.301708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.293695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.301708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.293811 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abd507-0528-408f-adc7-cdc1fa787d54-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.301708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.294805 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.301708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.295233 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.305782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.304028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.305782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.304552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcmlj\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-kube-api-access-lcmlj\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.305782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.304611 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.305782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.304673 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.305782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.304711 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.305782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.304794 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abd507-0528-408f-adc7-cdc1fa787d54-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.315782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.315372 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.319780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.316007 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.319780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.316264 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.319780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.316642 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.324498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.321387 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abd507-0528-408f-adc7-cdc1fa787d54-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.324799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.324777 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abd507-0528-408f-adc7-cdc1fa787d54-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.333707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.333682 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcmlj\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-kube-api-access-lcmlj\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.342398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.342370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.459599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.459571 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:31.609458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.609377 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:31.730563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.730526 4442 generic.go:356] "Generic (PLEG): container finished" podID="0ac811d5-6d49-4db7-a098-343417e96c71" containerID="1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3" exitCode=0 Oct 30 03:28:31.730725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.730580 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" Oct 30 03:28:31.730725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.730614 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" event={"ID":"0ac811d5-6d49-4db7-a098-343417e96c71","Type":"ContainerDied","Data":"1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3"} Oct 30 03:28:31.730725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.730668 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst" event={"ID":"0ac811d5-6d49-4db7-a098-343417e96c71","Type":"ContainerDied","Data":"2dc21d03d684aa1e7075d82c6c4a8d0fea100f7bbd53d5aa7750b2fa59fd70f9"} Oct 30 03:28:31.730725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.730689 4442 scope.go:122] "RemoveContainer" containerID="1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3" Oct 30 03:28:31.741045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.741021 4442 scope.go:122] "RemoveContainer" containerID="1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3" Oct 30 03:28:31.741340 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:31.741314 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3\": container with ID starting with 1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3 not found: ID does not exist" containerID="1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3" Oct 30 03:28:31.741434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.741346 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3"} err="failed to get container status \"1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3\": rpc error: code = NotFound desc = could not find container \"1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3\": container with ID starting with 1d8de86bc8f98c13132530979c0fc52d4909d8b66da11b43a97c03906582c3e3 not found: ID does not exist" Oct 30 03:28:31.770548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.770526 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn"] Oct 30 03:28:31.771946 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:28:31.771921 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10abd507_0528_408f_adc7_cdc1fa787d54.slice/crio-19779af1f3733f1adce6cf864b8a9f070a6b59becd03c35b9f5adec2fc0b7940 WatchSource:0}: Error finding container 19779af1f3733f1adce6cf864b8a9f070a6b59becd03c35b9f5adec2fc0b7940: Status 404 returned error can't find the container with id 19779af1f3733f1adce6cf864b8a9f070a6b59becd03c35b9f5adec2fc0b7940 Oct 30 03:28:31.774584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.774552 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:31.774676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.774628 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:31.774676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.774670 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:28:31.776561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.776538 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst"] Oct 30 03:28:31.779027 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:31.779008 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6c106c-5f564cb9cb-7lxst"] Oct 30 03:28:32.117227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:32.117186 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ac811d5-6d49-4db7-a098-343417e96c71" path="/var/lib/kubelet/pods/0ac811d5-6d49-4db7-a098-343417e96c71/volumes" Oct 30 03:28:32.739622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:32.739580 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" event={"ID":"10abd507-0528-408f-adc7-cdc1fa787d54","Type":"ContainerStarted","Data":"8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c"} Oct 30 03:28:32.739824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:32.739640 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" event={"ID":"10abd507-0528-408f-adc7-cdc1fa787d54","Type":"ContainerStarted","Data":"19779af1f3733f1adce6cf864b8a9f070a6b59becd03c35b9f5adec2fc0b7940"} Oct 30 03:28:32.816017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:32.815960 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podStartSLOduration=1.81594564 podStartE2EDuration="1.81594564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:32.803397685 +0000 UTC m=+1536.805750911" watchObservedRunningTime="2026-10-30 03:28:32.81594564 +0000 UTC m=+1536.818298836" Oct 30 03:28:33.463778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:33.462889 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 03:28:33.463778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:33.462963 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 03:28:33.463778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:33.463365 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:33.559713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:33.559679 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:34.460956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:34.460927 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 03:28:34.461111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:34.460976 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 03:28:35.457076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:35.457041 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:35.460132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:35.460110 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 03:28:35.460228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:35.460154 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 03:28:36.460310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:36.460275 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 03:28:36.460708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:36.460327 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 03:28:37.328560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:37.328526 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:37.460714 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:37.460676 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 03:28:37.461216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:37.460740 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 03:28:38.459926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:38.459895 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 03:28:38.460101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:38.459958 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 03:28:39.042151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:39.042122 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:39.945938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:39.945893 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:39.946292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:39.946266 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:39.947162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:39.947141 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:28:40.381100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:40.381065 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:40.483294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:40.483261 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:41.116931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:41.116888 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:41.353145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:41.353106 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:41.488150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:41.488062 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:42.253984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:42.253955 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:43.143112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:43.143081 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:44.926868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:44.926835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:46.304573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:46.304542 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:46.572305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:46.572228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:48.331765 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:48.331709 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:48.432026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:48.431998 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:49.047861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:49.047827 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:28:50.195780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:50.195295 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:51.825239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:51.825196 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:53.508413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:53.508366 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:54.855029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:54.854993 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl"] Oct 30 03:28:54.855548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:54.855217 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" podUID="e8d319f8-7558-4002-9911-cf2df2061b02" containerName="maas-api" containerID="cri-o://dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4" gracePeriod=30 Oct 30 03:28:54.872086 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:54.872051 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:28:54.904472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:54.904444 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:55.005588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.005560 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8"] Oct 30 03:28:55.005799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.005774 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" podUID="59f3c7cd-9c20-40b6-af33-1cf241af4f84" containerName="payload-processing" containerID="cri-o://510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02" gracePeriod=30 Oct 30 03:28:55.027214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.027186 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5"] Oct 30 03:28:55.027454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.027431 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" podUID="1fd1f205-ced9-46d3-a53d-d678b20d95f6" containerName="payload-pre-processing" containerID="cri-o://5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe" gracePeriod=30 Oct 30 03:28:55.090666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.086712 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:55.161820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.161795 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:28:55.181578 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.181489 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:55.277524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.277485 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d319f8-7558-4002-9911-cf2df2061b02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d319f8-7558-4002-9911-cf2df2061b02-openshift-service-ca\") pod \"e8d319f8-7558-4002-9911-cf2df2061b02\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " Oct 30 03:28:55.277706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.277588 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-maas-api-tls\") pod \"e8d319f8-7558-4002-9911-cf2df2061b02\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " Oct 30 03:28:55.277706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.277626 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d319f8-7558-4002-9911-cf2df2061b02-kube-api-access-sdgvp\" (UniqueName: \"kubernetes.io/projected/e8d319f8-7558-4002-9911-cf2df2061b02-kube-api-access-sdgvp\") pod \"e8d319f8-7558-4002-9911-cf2df2061b02\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " Oct 30 03:28:55.277903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.277727 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-metrics-certs\") pod \"e8d319f8-7558-4002-9911-cf2df2061b02\" (UID: \"e8d319f8-7558-4002-9911-cf2df2061b02\") " Oct 30 03:28:55.279111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.279079 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d319f8-7558-4002-9911-cf2df2061b02-openshift-service-ca" pod "e8d319f8-7558-4002-9911-cf2df2061b02" (UID: "e8d319f8-7558-4002-9911-cf2df2061b02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:55.285223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.284052 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-maas-api-tls" pod "e8d319f8-7558-4002-9911-cf2df2061b02" (UID: "e8d319f8-7558-4002-9911-cf2df2061b02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:55.285223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.284173 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d319f8-7558-4002-9911-cf2df2061b02-kube-api-access-sdgvp" pod "e8d319f8-7558-4002-9911-cf2df2061b02" (UID: "e8d319f8-7558-4002-9911-cf2df2061b02"). InnerVolumeSpecName "kube-api-access-sdgvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:55.285223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.284268 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-metrics-certs" pod "e8d319f8-7558-4002-9911-cf2df2061b02" (UID: "e8d319f8-7558-4002-9911-cf2df2061b02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:28:55.315356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.315320 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:28:55.384451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.380669 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.384451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.380704 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdgvp\" (UniqueName: \"kubernetes.io/projected/e8d319f8-7558-4002-9911-cf2df2061b02-kube-api-access-sdgvp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.384451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.380719 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d319f8-7558-4002-9911-cf2df2061b02-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.384451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.380732 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d319f8-7558-4002-9911-cf2df2061b02-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.392379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.390521 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:28:55.481391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.481353 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f3c7cd-9c20-40b6-af33-1cf241af4f84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f3c7cd-9c20-40b6-af33-1cf241af4f84-plugins-config-volume\") pod \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " Oct 30 03:28:55.481568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.481453 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f3c7cd-9c20-40b6-af33-1cf241af4f84-kube-api-access-2scdd\" (UniqueName: \"kubernetes.io/projected/59f3c7cd-9c20-40b6-af33-1cf241af4f84-kube-api-access-2scdd\") pod \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\" (UID: \"59f3c7cd-9c20-40b6-af33-1cf241af4f84\") " Oct 30 03:28:55.481741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.481716 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f3c7cd-9c20-40b6-af33-1cf241af4f84-plugins-config-volume" pod "59f3c7cd-9c20-40b6-af33-1cf241af4f84" (UID: "59f3c7cd-9c20-40b6-af33-1cf241af4f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:55.483637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.483614 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f3c7cd-9c20-40b6-af33-1cf241af4f84-kube-api-access-2scdd" pod "59f3c7cd-9c20-40b6-af33-1cf241af4f84" (UID: "59f3c7cd-9c20-40b6-af33-1cf241af4f84"). InnerVolumeSpecName "kube-api-access-2scdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:55.582985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.582953 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd1f205-ced9-46d3-a53d-d678b20d95f6-kube-api-access-g8l2l\" (UniqueName: \"kubernetes.io/projected/1fd1f205-ced9-46d3-a53d-d678b20d95f6-kube-api-access-g8l2l\") pod \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " Oct 30 03:28:55.583148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.583034 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd1f205-ced9-46d3-a53d-d678b20d95f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd1f205-ced9-46d3-a53d-d678b20d95f6-plugins-config-volume\") pod \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\" (UID: \"1fd1f205-ced9-46d3-a53d-d678b20d95f6\") " Oct 30 03:28:55.583317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.583302 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59f3c7cd-9c20-40b6-af33-1cf241af4f84-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.583317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.583316 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2scdd\" (UniqueName: \"kubernetes.io/projected/59f3c7cd-9c20-40b6-af33-1cf241af4f84-kube-api-access-2scdd\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.583437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.583398 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd1f205-ced9-46d3-a53d-d678b20d95f6-plugins-config-volume" pod "1fd1f205-ced9-46d3-a53d-d678b20d95f6" (UID: "1fd1f205-ced9-46d3-a53d-d678b20d95f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:28:55.585262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.585239 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd1f205-ced9-46d3-a53d-d678b20d95f6-kube-api-access-g8l2l" pod "1fd1f205-ced9-46d3-a53d-d678b20d95f6" (UID: "1fd1f205-ced9-46d3-a53d-d678b20d95f6"). InnerVolumeSpecName "kube-api-access-g8l2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:28:55.646043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.645966 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:55.684605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.684570 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8l2l\" (UniqueName: \"kubernetes.io/projected/1fd1f205-ced9-46d3-a53d-d678b20d95f6-kube-api-access-g8l2l\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.684605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.684598 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd1f205-ced9-46d3-a53d-d678b20d95f6-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:28:55.938341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.938259 4442 generic.go:356] "Generic (PLEG): container finished" podID="1fd1f205-ced9-46d3-a53d-d678b20d95f6" containerID="5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe" exitCode=0 Oct 30 03:28:55.938341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.938321 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" Oct 30 03:28:55.938876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.938344 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" event={"ID":"1fd1f205-ced9-46d3-a53d-d678b20d95f6","Type":"ContainerDied","Data":"5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe"} Oct 30 03:28:55.938876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.938397 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5" event={"ID":"1fd1f205-ced9-46d3-a53d-d678b20d95f6","Type":"ContainerDied","Data":"bb651072eef9417e3f4fe9c9a208a9d38e8674eca45265821eb91b7f4bbc2b9d"} Oct 30 03:28:55.938876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.938418 4442 scope.go:122] "RemoveContainer" containerID="5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe" Oct 30 03:28:55.939895 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.939876 4442 generic.go:356] "Generic (PLEG): container finished" podID="59f3c7cd-9c20-40b6-af33-1cf241af4f84" containerID="510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02" exitCode=0 Oct 30 03:28:55.939997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.939903 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" event={"ID":"59f3c7cd-9c20-40b6-af33-1cf241af4f84","Type":"ContainerDied","Data":"510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02"} Oct 30 03:28:55.939997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.939936 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" Oct 30 03:28:55.940092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.939945 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8" event={"ID":"59f3c7cd-9c20-40b6-af33-1cf241af4f84","Type":"ContainerDied","Data":"7821bb1799198bc12210cdb3f21182b40079ff37a263415bb9766f233425b3c4"} Oct 30 03:28:55.941429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.941407 4442 generic.go:356] "Generic (PLEG): container finished" podID="e8d319f8-7558-4002-9911-cf2df2061b02" containerID="dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4" exitCode=0 Oct 30 03:28:55.941546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.941436 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" event={"ID":"e8d319f8-7558-4002-9911-cf2df2061b02","Type":"ContainerDied","Data":"dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4"} Oct 30 03:28:55.941546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.941454 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" Oct 30 03:28:55.941546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.941474 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl" event={"ID":"e8d319f8-7558-4002-9911-cf2df2061b02","Type":"ContainerDied","Data":"9c81168e2d4fff4b30f346425ab8c02f85c7bc94c8e794f5aab373296eca8b9a"} Oct 30 03:28:55.950358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.950337 4442 scope.go:122] "RemoveContainer" containerID="5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe" Oct 30 03:28:55.950898 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:55.950793 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe\": container with ID starting with 5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe not found: ID does not exist" containerID="5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe" Oct 30 03:28:55.950898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.950816 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe"} err="failed to get container status \"5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe\": rpc error: code = NotFound desc = could not find container \"5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe\": container with ID starting with 5546ea92b63002f0dd3bd30c4cad01ddce821449d4c88c0a50ab4c61ef2dcfbe not found: ID does not exist" Oct 30 03:28:55.950898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.950834 4442 scope.go:122] "RemoveContainer" containerID="510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02" Oct 30 03:28:55.959825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.959800 4442 scope.go:122] "RemoveContainer" containerID="510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02" Oct 30 03:28:55.960116 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:55.960096 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02\": container with ID starting with 510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02 not found: ID does not exist" containerID="510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02" Oct 30 03:28:55.960175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.960123 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02"} err="failed to get container status \"510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02\": rpc error: code = NotFound desc = could not find container \"510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02\": container with ID starting with 510b88a21af63f4e283e29446c9bb76f3f763de5e317a3b7ad4c3202ba70cd02 not found: ID does not exist" Oct 30 03:28:55.960175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.960140 4442 scope.go:122] "RemoveContainer" containerID="dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4" Oct 30 03:28:55.968461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.968447 4442 scope.go:122] "RemoveContainer" containerID="dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4" Oct 30 03:28:55.968711 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:28:55.968695 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4\": container with ID starting with dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4 not found: ID does not exist" containerID="dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4" Oct 30 03:28:55.968773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:55.968717 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4"} err="failed to get container status \"dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4\": rpc error: code = NotFound desc = could not find container \"dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4\": container with ID starting with dd8b3a57520565064aabbcad974871545a5ceb8c837eeedd4010db0f89a1e1c4 not found: ID does not exist" Oct 30 03:28:56.010611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.010589 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5"] Oct 30 03:28:56.016599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.016565 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-efbbc5-7698f8c7cf-mntw5"] Oct 30 03:28:56.044457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.044435 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl"] Oct 30 03:28:56.047594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.047575 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-efbbc5-7cc8bcc665-dkdxl"] Oct 30 03:28:56.081186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.081160 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8"] Oct 30 03:28:56.090039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.090018 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-efbbc5-64b4965578-q8pr8"] Oct 30 03:28:56.107136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.107115 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd1f205-ced9-46d3-a53d-d678b20d95f6" path="/var/lib/kubelet/pods/1fd1f205-ced9-46d3-a53d-d678b20d95f6/volumes" Oct 30 03:28:56.107492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.107477 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f3c7cd-9c20-40b6-af33-1cf241af4f84" path="/var/lib/kubelet/pods/59f3c7cd-9c20-40b6-af33-1cf241af4f84/volumes" Oct 30 03:28:56.107872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:56.107859 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d319f8-7558-4002-9911-cf2df2061b02" path="/var/lib/kubelet/pods/e8d319f8-7558-4002-9911-cf2df2061b02/volumes" Oct 30 03:28:57.399813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:57.399784 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:28:59.106298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:28:59.106265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:00.796450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:00.796417 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:01.224792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.224692 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6"] Oct 30 03:29:01.225193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225177 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e8d319f8-7558-4002-9911-cf2df2061b02" containerName="maas-api" Oct 30 03:29:01.225236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225194 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d319f8-7558-4002-9911-cf2df2061b02" containerName="maas-api" Oct 30 03:29:01.225236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225225 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1fd1f205-ced9-46d3-a53d-d678b20d95f6" containerName="payload-pre-processing" Oct 30 03:29:01.225236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225230 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd1f205-ced9-46d3-a53d-d678b20d95f6" containerName="payload-pre-processing" Oct 30 03:29:01.225236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225236 4442 cpu_manager.go:414] "containerMap: removing container" podUID="59f3c7cd-9c20-40b6-af33-1cf241af4f84" containerName="payload-processing" Oct 30 03:29:01.225358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225241 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f3c7cd-9c20-40b6-af33-1cf241af4f84" containerName="payload-processing" Oct 30 03:29:01.225358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225338 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd1f205-ced9-46d3-a53d-d678b20d95f6" containerName="payload-pre-processing" Oct 30 03:29:01.225358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225351 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f3c7cd-9c20-40b6-af33-1cf241af4f84" containerName="payload-processing" Oct 30 03:29:01.225469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.225364 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d319f8-7558-4002-9911-cf2df2061b02" containerName="maas-api" Oct 30 03:29:01.233088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.233065 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.261733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.261702 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6"] Oct 30 03:29:01.338868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.338824 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.338877 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.338996 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.339027 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.339101 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.339183 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv447\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-kube-api-access-kv447\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.339225 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.339330 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.339379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.339365 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440368 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440410 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440600 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440637 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440666 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440800 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440833 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440860 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv447\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-kube-api-access-kv447\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.440966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440889 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.441070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.440974 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.441070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.441043 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-credential-socket\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.441201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.441177 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-certs\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.441258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.441195 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-data\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.441297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.441256 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istiod-ca-cert\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.443113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.443087 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-envoy\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.443272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.443254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-podinfo\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.458496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.458467 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-token\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.458592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.458515 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv447\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-kube-api-access-kv447\") pod \"e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.548408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.548379 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:01.858277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.858251 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6"] Oct 30 03:29:01.859372 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:29:01.859345 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f9e7b0_9769_4a43_b2b5_8ab40addd8c0.slice/crio-9f2d0cc060cd43bdae07d38d06c029424369cd83b554cf95d3288f090572d54e WatchSource:0}: Error finding container 9f2d0cc060cd43bdae07d38d06c029424369cd83b554cf95d3288f090572d54e: Status 404 returned error can't find the container with id 9f2d0cc060cd43bdae07d38d06c029424369cd83b554cf95d3288f090572d54e Oct 30 03:29:01.861820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.861790 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:29:01.861957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.861874 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:29:01.861957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.861921 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:29:01.974273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.974229 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" event={"ID":"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0","Type":"ContainerStarted","Data":"c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898"} Oct 30 03:29:01.974433 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:01.974301 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" event={"ID":"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0","Type":"ContainerStarted","Data":"9f2d0cc060cd43bdae07d38d06c029424369cd83b554cf95d3288f090572d54e"} Oct 30 03:29:02.015993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:02.015906 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podStartSLOduration=1.015888183 podStartE2EDuration="1.015888183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:02.015345009 +0000 UTC m=+1566.017698207" watchObservedRunningTime="2026-10-30 03:29:02.015888183 +0000 UTC m=+1566.018241384" Oct 30 03:29:02.514172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:02.514136 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:02.548825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:02.548791 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:02.550515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:02.550491 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 03:29:02.550634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:02.550535 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 03:29:02.654128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:02.654095 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:02.779936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:02.779862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:03.372101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:03.372070 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:03.549669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:03.549638 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 03:29:03.549893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:03.549688 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 03:29:03.765973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:03.765943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:04.445270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:04.445233 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:04.549224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:04.549182 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 03:29:04.549403 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:04.549252 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 03:29:05.275670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:05.275635 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:05.549373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:05.549278 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 03:29:05.549373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:05.549342 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 03:29:05.737903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:05.737852 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:06.158047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:06.158016 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:06.408437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:06.408344 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:06.550774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:06.550111 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 03:29:06.550774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:06.550176 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 03:29:07.551777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:07.551193 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 03:29:07.551777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:07.551254 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 03:29:08.399428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:08.399393 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:08.549500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:08.549468 4442 patch_prober.go:29] interesting pod/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 30 03:29:08.549671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:08.549522 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 30 03:29:09.605282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:09.605203 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:09.713658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:09.713622 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:09.990908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:09.990820 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:10.087099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:10.087060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:10.307423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:10.307390 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:10.307721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:10.307696 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:10.309838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:10.309818 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:29:10.428385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:10.428352 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:11.306085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.306052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:11.406116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.406085 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:11.489233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.489206 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp"] Oct 30 03:29:11.494057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.494041 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:11.510388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.510364 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp"] Oct 30 03:29:11.518584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.518551 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-65f21d\"" Oct 30 03:29:11.518707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.518621 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-65f21d-dockercfg-rtsgh\"" Oct 30 03:29:11.665100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.665011 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b8d439-4925-4ca6-87a2-f5514e9f834b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:11.665802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.665421 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/46b8d439-4925-4ca6-87a2-f5514e9f834b-kube-api-access-6wq6c\") pod \"payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:11.766669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.766638 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b8d439-4925-4ca6-87a2-f5514e9f834b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:11.766848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.766703 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/46b8d439-4925-4ca6-87a2-f5514e9f834b-kube-api-access-6wq6c\") pod \"payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:11.767419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.767399 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b8d439-4925-4ca6-87a2-f5514e9f834b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:11.784675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.784644 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/46b8d439-4925-4ca6-87a2-f5514e9f834b-kube-api-access-6wq6c\") pod \"payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:11.807999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:11.807975 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:12.107811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:12.107787 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp"] Oct 30 03:29:12.108722 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:29:12.108688 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b8d439_4925_4ca6_87a2_f5514e9f834b.slice/crio-a44b2c399738eee34c2edcee2b061fa945109f03afae2e5d084cc89c53177c05 WatchSource:0}: Error finding container a44b2c399738eee34c2edcee2b061fa945109f03afae2e5d084cc89c53177c05: Status 404 returned error can't find the container with id a44b2c399738eee34c2edcee2b061fa945109f03afae2e5d084cc89c53177c05 Oct 30 03:29:12.174126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:12.174101 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:12.906248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:12.906215 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l"] Oct 30 03:29:12.911020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:12.911001 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:12.938548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:12.938521 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l"] Oct 30 03:29:13.072500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.072461 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" event={"ID":"46b8d439-4925-4ca6-87a2-f5514e9f834b","Type":"ContainerStarted","Data":"d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd"} Oct 30 03:29:13.072677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.072548 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" event={"ID":"46b8d439-4925-4ca6-87a2-f5514e9f834b","Type":"ContainerStarted","Data":"a44b2c399738eee34c2edcee2b061fa945109f03afae2e5d084cc89c53177c05"} Oct 30 03:29:13.072677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.072604 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:13.082277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082251 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082286 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082328 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082398 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwl92\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-kube-api-access-mwl92\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082516 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082595 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ad860-7156-451c-9b66-001c8bbd7107-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082619 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082719 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.082882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.082856 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.108366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.108325 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" podStartSLOduration=2.108314297 podStartE2EDuration="2.108314297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:13.10729736 +0000 UTC m=+1577.109650557" watchObservedRunningTime="2026-10-30 03:29:13.108314297 +0000 UTC m=+1577.110667486" Oct 30 03:29:13.183499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183424 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.183499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183468 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ad860-7156-451c-9b66-001c8bbd7107-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.183499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183498 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.183783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183595 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.183783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183648 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.183783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.183783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183725 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.184139 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.184114 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.184221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.184124 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-certs\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.184221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.183848 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.184340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.184277 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwl92\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-kube-api-access-mwl92\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.184340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.184280 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ad860-7156-451c-9b66-001c8bbd7107-istiod-ca-cert\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.184471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.184450 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-credential-socket\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.184559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.184533 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-data\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.186139 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.186120 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-podinfo\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.186246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.186186 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-envoy\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.201317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.201293 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-token\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.201710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.201689 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwl92\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-kube-api-access-mwl92\") pod \"e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.230924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.230873 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:13.570627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.570597 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l"] Oct 30 03:29:13.571574 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:29:13.571549 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8ad860_7156_451c_9b66_001c8bbd7107.slice/crio-17af3f461c2594eb0e970b7b85672c4bdcce3f75b28a704453e3801b4861c074 WatchSource:0}: Error finding container 17af3f461c2594eb0e970b7b85672c4bdcce3f75b28a704453e3801b4861c074: Status 404 returned error can't find the container with id 17af3f461c2594eb0e970b7b85672c4bdcce3f75b28a704453e3801b4861c074 Oct 30 03:29:13.574998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.574969 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:29:13.575084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.575034 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:29:13.575084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.575062 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:29:13.840155 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:13.840058 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:14.080184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:14.080146 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" event={"ID":"ee8ad860-7156-451c-9b66-001c8bbd7107","Type":"ContainerStarted","Data":"b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c"} Oct 30 03:29:14.080588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:14.080209 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" event={"ID":"ee8ad860-7156-451c-9b66-001c8bbd7107","Type":"ContainerStarted","Data":"17af3f461c2594eb0e970b7b85672c4bdcce3f75b28a704453e3801b4861c074"} Oct 30 03:29:14.141252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:14.141164 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podStartSLOduration=2.141152416 podStartE2EDuration="2.141152416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:14.128948945 +0000 UTC m=+1578.131302143" watchObservedRunningTime="2026-10-30 03:29:14.141152416 +0000 UTC m=+1578.143505612" Oct 30 03:29:14.231891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:14.231862 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:14.233487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:14.233462 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 03:29:14.233631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:14.233525 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 03:29:15.231541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:15.231505 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 03:29:15.231978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:15.231558 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 03:29:15.357674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:15.357639 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:29:15.644799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:15.644748 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:16.232011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:16.231975 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 03:29:16.232418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:16.232034 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 03:29:17.231303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:17.231263 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 03:29:17.231522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:17.231322 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 03:29:17.338981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:17.338944 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:18.232072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:18.232028 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 03:29:18.232387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:18.232088 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 03:29:19.113618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:19.113584 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:19.232326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:19.232279 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 03:29:19.232515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:19.232357 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 03:29:20.232241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:20.232202 4442 patch_prober.go:29] interesting pod/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 03:29:20.232626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:20.232271 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 03:29:20.898544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:20.898512 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:21.236405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:21.236326 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:21.236801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:21.236727 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:21.237354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:21.237339 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:29:22.510522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.510488 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j"] Oct 30 03:29:22.518797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.516724 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.532988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.532960 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j"] Oct 30 03:29:22.533186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.533171 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-65f21d-serving-cert\"" Oct 30 03:29:22.593806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.593739 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:22.644879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.644841 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66580b9-aa52-4b09-8be4-77e00592db41-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.645035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.644884 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfxw\" (UniqueName: \"kubernetes.io/projected/c66580b9-aa52-4b09-8be4-77e00592db41-kube-api-access-mmfxw\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.645083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.645042 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-metrics-certs\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.645126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.645086 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.680108 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.680075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:22.746090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.746053 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.746285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.746156 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66580b9-aa52-4b09-8be4-77e00592db41-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.746285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.746187 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfxw\" (UniqueName: \"kubernetes.io/projected/c66580b9-aa52-4b09-8be4-77e00592db41-kube-api-access-mmfxw\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.746285 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:29:22.746229 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-serving-cert: secret "maas-api-e2e-praxis-a-65f21d-serving-cert" not found Oct 30 03:29:22.746438 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:29:22.746302 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls podName:c66580b9-aa52-4b09-8be4-77e00592db41 nodeName:}" failed. No retries permitted until 2026-10-30 03:29:23.246282457 +0000 UTC m=+1587.248635645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls") pod "maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" (UID: "c66580b9-aa52-4b09-8be4-77e00592db41") : secret "maas-api-e2e-praxis-a-65f21d-serving-cert" not found Oct 30 03:29:22.746438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.746391 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-metrics-certs\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.748126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.748103 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66580b9-aa52-4b09-8be4-77e00592db41-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.750671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.750645 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-metrics-certs\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.764519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.764462 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfxw\" (UniqueName: \"kubernetes.io/projected/c66580b9-aa52-4b09-8be4-77e00592db41-kube-api-access-mmfxw\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:22.858589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:22.858560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:23.252059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:23.251870 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:23.254391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:23.254367 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls\") pod \"maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:23.437365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:23.437335 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:23.754548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:23.754524 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j"] Oct 30 03:29:23.755917 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:29:23.755890 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66580b9_aa52_4b09_8be4_77e00592db41.slice/crio-45d9d6a901877ded860dae19455791952692ef52ce1fbf5302c4c72654d685a7 WatchSource:0}: Error finding container 45d9d6a901877ded860dae19455791952692ef52ce1fbf5302c4c72654d685a7: Status 404 returned error can't find the container with id 45d9d6a901877ded860dae19455791952692ef52ce1fbf5302c4c72654d685a7 Oct 30 03:29:23.757229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:23.757212 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:29:24.083591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:24.083567 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:29:24.160418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:24.160382 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" event={"ID":"c66580b9-aa52-4b09-8be4-77e00592db41","Type":"ContainerStarted","Data":"a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7"} Oct 30 03:29:24.160545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:24.160451 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" event={"ID":"c66580b9-aa52-4b09-8be4-77e00592db41","Type":"ContainerStarted","Data":"45d9d6a901877ded860dae19455791952692ef52ce1fbf5302c4c72654d685a7"} Oct 30 03:29:24.409074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:24.408952 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:24.863834 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:29:24.863799 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:29:25.166540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:25.166463 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:25.204310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:25.204250 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" podStartSLOduration=2.890188487 podStartE2EDuration="3.204237185s" totalImagesPullingTime="314.048698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:25.202105598 +0000 UTC m=+1589.204458794" watchObservedRunningTime="2026-10-30 03:29:25.204237185 +0000 UTC m=+1589.206590381" Oct 30 03:29:26.113818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:26.113786 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:27.820979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:27.820941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:29.412069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:29.412038 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:31.108905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:31.108876 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:31.205512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:31.205472 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:29:32.709990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:32.709956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:34.525094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:34.525059 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:34.904711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:34.904609 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:29:36.387915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:36.387880 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:37.907548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:37.907514 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:39.468996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:39.468965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:39.569147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:39.569115 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:41.311038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:41.311004 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:42.973134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:42.973100 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:44.499988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:44.499962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:44.717871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:44.717838 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:45.050511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:45.050473 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:46.227349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:46.227309 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:46.339179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:46.339150 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:46.449408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:46.449375 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:46.653170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:46.653127 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:46.741437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:46.741408 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:48.024347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:48.024319 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:48.465501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:48.465395 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:49.812355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:49.812318 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:50.084156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:50.084069 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:50.185807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:50.185776 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:51.722074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:51.722033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:51.829952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:51.829905 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:51.932147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:51.932118 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:51.948305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:51.948274 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6"] Oct 30 03:29:51.952008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:51.951992 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:51.965880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:51.965853 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-773fbb\"" Oct 30 03:29:51.968844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:51.968819 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6"] Oct 30 03:29:52.015697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.015621 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-773fbb-dockercfg-ds2ld\"" Oct 30 03:29:52.040592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.040546 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvf6f\" (UniqueName: \"kubernetes.io/projected/a384c723-3eb5-431c-9955-255cf34fdb05-kube-api-access-qvf6f\") pod \"payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:52.044877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.044842 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a384c723-3eb5-431c-9955-255cf34fdb05-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:52.112457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.112414 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd"] Oct 30 03:29:52.120678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.120637 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd"] Oct 30 03:29:52.121008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.120805 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.170920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.170888 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvf6f\" (UniqueName: \"kubernetes.io/projected/a384c723-3eb5-431c-9955-255cf34fdb05-kube-api-access-qvf6f\") pod \"payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:52.171112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.170930 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a384c723-3eb5-431c-9955-255cf34fdb05-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:52.171641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.171621 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a384c723-3eb5-431c-9955-255cf34fdb05-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:52.189064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.189039 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvf6f\" (UniqueName: \"kubernetes.io/projected/a384c723-3eb5-431c-9955-255cf34fdb05-kube-api-access-qvf6f\") pod \"payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:52.265387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.265360 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:52.272122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.272095 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69h4x\" (UniqueName: \"kubernetes.io/projected/1422630a-fd28-4209-9316-586b74125bce-kube-api-access-69h4x\") pod \"payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.272257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.272131 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1422630a-fd28-4209-9316-586b74125bce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.373047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.373012 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69h4x\" (UniqueName: \"kubernetes.io/projected/1422630a-fd28-4209-9316-586b74125bce-kube-api-access-69h4x\") pod \"payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.373236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.373060 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1422630a-fd28-4209-9316-586b74125bce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.373845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.373825 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1422630a-fd28-4209-9316-586b74125bce-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.391168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.391136 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69h4x\" (UniqueName: \"kubernetes.io/projected/1422630a-fd28-4209-9316-586b74125bce-kube-api-access-69h4x\") pod \"payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.458907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.458883 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:52.568639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.568602 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6"] Oct 30 03:29:52.570763 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:29:52.570725 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda384c723_3eb5_431c_9955_255cf34fdb05.slice/crio-0574b374a7f70ec7565c1064f57fcdd8bd0705bbd6124892b79705704726fa31 WatchSource:0}: Error finding container 0574b374a7f70ec7565c1064f57fcdd8bd0705bbd6124892b79705704726fa31: Status 404 returned error can't find the container with id 0574b374a7f70ec7565c1064f57fcdd8bd0705bbd6124892b79705704726fa31 Oct 30 03:29:52.760681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:52.760654 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd"] Oct 30 03:29:52.761080 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:29:52.760826 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1422630a_fd28_4209_9316_586b74125bce.slice/crio-66232c5a1aa5d8588c084116dc6a01bab70443ae445bbbfed1e26d430d690f55 WatchSource:0}: Error finding container 66232c5a1aa5d8588c084116dc6a01bab70443ae445bbbfed1e26d430d690f55: Status 404 returned error can't find the container with id 66232c5a1aa5d8588c084116dc6a01bab70443ae445bbbfed1e26d430d690f55 Oct 30 03:29:53.347001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.346965 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" event={"ID":"1422630a-fd28-4209-9316-586b74125bce","Type":"ContainerStarted","Data":"22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f"} Oct 30 03:29:53.347180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.347022 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" event={"ID":"1422630a-fd28-4209-9316-586b74125bce","Type":"ContainerStarted","Data":"66232c5a1aa5d8588c084116dc6a01bab70443ae445bbbfed1e26d430d690f55"} Oct 30 03:29:53.347180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.347051 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:29:53.348500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.348477 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" event={"ID":"a384c723-3eb5-431c-9955-255cf34fdb05","Type":"ContainerStarted","Data":"bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408"} Oct 30 03:29:53.348634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.348513 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" event={"ID":"a384c723-3eb5-431c-9955-255cf34fdb05","Type":"ContainerStarted","Data":"0574b374a7f70ec7565c1064f57fcdd8bd0705bbd6124892b79705704726fa31"} Oct 30 03:29:53.348634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.348564 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:29:53.385428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.385380 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" podStartSLOduration=1.385363632 podStartE2EDuration="1.385363632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:53.383547226 +0000 UTC m=+1617.385900434" watchObservedRunningTime="2026-10-30 03:29:53.385363632 +0000 UTC m=+1617.387716832" Oct 30 03:29:53.418897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.418873 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:53.423648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:53.423603 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" podStartSLOduration=2.423587476 podStartE2EDuration="2.423587476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:29:53.423156219 +0000 UTC m=+1617.425509417" watchObservedRunningTime="2026-10-30 03:29:53.423587476 +0000 UTC m=+1617.425940677" Oct 30 03:29:54.861353 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:29:54.861318 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:29:55.003745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:55.003713 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:56.686824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:56.686785 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:29:58.314654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:29:58.314617 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:00.082385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:00.082350 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:01.717681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:01.717641 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:03.334562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:03.334531 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:04.356679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:04.356647 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:30:04.357085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:04.356736 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:30:05.044811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:05.044776 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:05.142560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:05.142530 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:05.243462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:05.243429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:06.826030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:06.825995 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:08.412138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:08.412109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:10.112363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:10.112335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:11.717586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:11.717547 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:13.296678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:13.296644 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:14.021780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:14.021722 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:30:15.113601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:15.113562 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:16.814251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:16.814190 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:18.436474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:18.436442 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:20.077256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:20.077223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:20.745958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:20.745924 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:20.852610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:20.852574 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:22.080287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:22.080254 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:22.181347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:22.181314 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:23.100846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:23.100812 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:23.905189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:23.905148 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:24.861462 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:30:24.861429 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:30:25.491380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:25.491326 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:25.710156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:25.710118 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:26.608597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:26.608553 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:27.291741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:27.291705 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:27.867275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:27.867243 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:29.028255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:29.028227 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:29.381541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:29.381459 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:30.637237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:30.637206 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:32.081362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:32.081332 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:30:32.364022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:32.363939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:33.921559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:33.921528 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:34.015406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:34.015372 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp"] Oct 30 03:30:34.017509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:34.017478 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:34.111580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:34.111541 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:34.212992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:34.212796 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:35.380793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:35.380744 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:35.559299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:35.559265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:36.302228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:36.302191 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:37.875860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:37.875830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:39.015833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.015798 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" podUID="46b8d439-4925-4ca6-87a2-f5514e9f834b" containerName="payload-pre-processing" containerID="cri-o://d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd" gracePeriod=25 Oct 30 03:30:39.285118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.285054 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:30:39.383639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.383609 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b8d439-4925-4ca6-87a2-f5514e9f834b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b8d439-4925-4ca6-87a2-f5514e9f834b-plugins-config-volume\") pod \"46b8d439-4925-4ca6-87a2-f5514e9f834b\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " Oct 30 03:30:39.383825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.383656 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b8d439-4925-4ca6-87a2-f5514e9f834b-kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/46b8d439-4925-4ca6-87a2-f5514e9f834b-kube-api-access-6wq6c\") pod \"46b8d439-4925-4ca6-87a2-f5514e9f834b\" (UID: \"46b8d439-4925-4ca6-87a2-f5514e9f834b\") " Oct 30 03:30:39.384018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.383994 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b8d439-4925-4ca6-87a2-f5514e9f834b-plugins-config-volume" pod "46b8d439-4925-4ca6-87a2-f5514e9f834b" (UID: "46b8d439-4925-4ca6-87a2-f5514e9f834b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:39.385896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.385867 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b8d439-4925-4ca6-87a2-f5514e9f834b-kube-api-access-6wq6c" pod "46b8d439-4925-4ca6-87a2-f5514e9f834b" (UID: "46b8d439-4925-4ca6-87a2-f5514e9f834b"). InnerVolumeSpecName "kube-api-access-6wq6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:39.485306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.485276 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wq6c\" (UniqueName: \"kubernetes.io/projected/46b8d439-4925-4ca6-87a2-f5514e9f834b-kube-api-access-6wq6c\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:39.485306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.485302 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46b8d439-4925-4ca6-87a2-f5514e9f834b-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:39.522169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.522144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:39.716549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.716455 4442 generic.go:356] "Generic (PLEG): container finished" podID="46b8d439-4925-4ca6-87a2-f5514e9f834b" containerID="d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd" exitCode=0 Oct 30 03:30:39.716816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.716793 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" event={"ID":"46b8d439-4925-4ca6-87a2-f5514e9f834b","Type":"ContainerDied","Data":"d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd"} Oct 30 03:30:39.717009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.716993 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" event={"ID":"46b8d439-4925-4ca6-87a2-f5514e9f834b","Type":"ContainerDied","Data":"a44b2c399738eee34c2edcee2b061fa945109f03afae2e5d084cc89c53177c05"} Oct 30 03:30:39.717897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.717871 4442 scope.go:122] "RemoveContainer" containerID="d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd" Oct 30 03:30:39.718017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.717996 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp" Oct 30 03:30:39.733698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.733681 4442 scope.go:122] "RemoveContainer" containerID="d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd" Oct 30 03:30:39.734038 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:30:39.734014 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd\": container with ID starting with d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd not found: ID does not exist" containerID="d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd" Oct 30 03:30:39.734089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.734046 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd"} err="failed to get container status \"d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd\": rpc error: code = NotFound desc = could not find container \"d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd\": container with ID starting with d018b134b13ddc72bafa1e04e5bbdfd55a97acd60c55888b682457b7d19310bd not found: ID does not exist" Oct 30 03:30:39.765149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.765129 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp"] Oct 30 03:30:39.776196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.776175 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-65f21d-8477fc7bcf-n92qp"] Oct 30 03:30:39.781652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:39.781633 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:40.107396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:40.107364 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b8d439-4925-4ca6-87a2-f5514e9f834b" path="/var/lib/kubelet/pods/46b8d439-4925-4ca6-87a2-f5514e9f834b/volumes" Oct 30 03:30:40.888736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:40.888708 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:40.986869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:40.986840 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:41.232046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:41.231967 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:41.573656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:41.573625 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:42.060392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:42.060360 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:42.827657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:42.827622 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j"] Oct 30 03:30:42.828194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:42.827895 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" podUID="c66580b9-aa52-4b09-8be4-77e00592db41" containerName="maas-api" containerID="cri-o://a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7" gracePeriod=30 Oct 30 03:30:42.847394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:42.847361 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:42.949863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:42.949831 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:43.141022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.140605 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:30:43.268769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.268724 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-metrics-certs\") pod \"c66580b9-aa52-4b09-8be4-77e00592db41\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " Oct 30 03:30:43.268961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.268831 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c66580b9-aa52-4b09-8be4-77e00592db41-kube-api-access-mmfxw\" (UniqueName: \"kubernetes.io/projected/c66580b9-aa52-4b09-8be4-77e00592db41-kube-api-access-mmfxw\") pod \"c66580b9-aa52-4b09-8be4-77e00592db41\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " Oct 30 03:30:43.268961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.268855 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c66580b9-aa52-4b09-8be4-77e00592db41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66580b9-aa52-4b09-8be4-77e00592db41-openshift-service-ca\") pod \"c66580b9-aa52-4b09-8be4-77e00592db41\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " Oct 30 03:30:43.268961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.268947 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls\") pod \"c66580b9-aa52-4b09-8be4-77e00592db41\" (UID: \"c66580b9-aa52-4b09-8be4-77e00592db41\") " Oct 30 03:30:43.269349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.269310 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66580b9-aa52-4b09-8be4-77e00592db41-openshift-service-ca" pod "c66580b9-aa52-4b09-8be4-77e00592db41" (UID: "c66580b9-aa52-4b09-8be4-77e00592db41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:43.271246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.271223 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls" pod "c66580b9-aa52-4b09-8be4-77e00592db41" (UID: "c66580b9-aa52-4b09-8be4-77e00592db41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:43.271653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.271630 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-metrics-certs" pod "c66580b9-aa52-4b09-8be4-77e00592db41" (UID: "c66580b9-aa52-4b09-8be4-77e00592db41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:43.271746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.271653 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66580b9-aa52-4b09-8be4-77e00592db41-kube-api-access-mmfxw" pod "c66580b9-aa52-4b09-8be4-77e00592db41" (UID: "c66580b9-aa52-4b09-8be4-77e00592db41"). InnerVolumeSpecName "kube-api-access-mmfxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:43.374949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.371623 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfxw\" (UniqueName: \"kubernetes.io/projected/c66580b9-aa52-4b09-8be4-77e00592db41-kube-api-access-mmfxw\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:43.374949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.371659 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66580b9-aa52-4b09-8be4-77e00592db41-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:43.374949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.371673 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:43.374949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.371685 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66580b9-aa52-4b09-8be4-77e00592db41-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:43.642134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.642104 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:43.741333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.741300 4442 generic.go:356] "Generic (PLEG): container finished" podID="c66580b9-aa52-4b09-8be4-77e00592db41" containerID="a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7" exitCode=0 Oct 30 03:30:43.741516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.741344 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" Oct 30 03:30:43.741516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.741353 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" event={"ID":"c66580b9-aa52-4b09-8be4-77e00592db41","Type":"ContainerDied","Data":"a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7"} Oct 30 03:30:43.741516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.741401 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j" event={"ID":"c66580b9-aa52-4b09-8be4-77e00592db41","Type":"ContainerDied","Data":"45d9d6a901877ded860dae19455791952692ef52ce1fbf5302c4c72654d685a7"} Oct 30 03:30:43.741516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.741422 4442 scope.go:122] "RemoveContainer" containerID="a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7" Oct 30 03:30:43.751468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.751451 4442 scope.go:122] "RemoveContainer" containerID="a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7" Oct 30 03:30:43.751724 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:30:43.751706 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7\": container with ID starting with a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7 not found: ID does not exist" containerID="a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7" Oct 30 03:30:43.751841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.751731 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7"} err="failed to get container status \"a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7\": rpc error: code = NotFound desc = could not find container \"a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7\": container with ID starting with a1d293c863da978e57e8ec1472dc9b42be1957b3495b829df6f52fa7e0c12ec7 not found: ID does not exist" Oct 30 03:30:43.788076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.788052 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j"] Oct 30 03:30:43.798670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:43.798649 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-65f21d-55cdd466f7-vv27j"] Oct 30 03:30:44.113353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:44.113319 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c66580b9-aa52-4b09-8be4-77e00592db41" path="/var/lib/kubelet/pods/c66580b9-aa52-4b09-8be4-77e00592db41/volumes" Oct 30 03:30:45.296197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:45.296160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:45.411762 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:45.411720 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:45.514945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:45.514912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:46.099810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:46.099774 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:46.514091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:46.514060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:46.624315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:46.624275 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:47.152581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:47.152551 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:48.402783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:48.402728 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr"] Oct 30 03:30:48.459029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:48.459001 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7"] Oct 30 03:30:48.561287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:48.561253 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:48.673382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:48.673290 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:48.879197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:48.879168 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:49.003611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:49.003581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:50.700187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:50.700091 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:52.333675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:52.333649 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:52.628276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:52.628201 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:53.403148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.403110 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" podUID="dfa35cf8-883a-494f-b949-41cd36f73b41" containerName="payload-processing" containerID="cri-o://12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7" gracePeriod=25 Oct 30 03:30:53.459805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.459774 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" podUID="fbbdb317-95af-4141-8f1b-55d7eff61a27" containerName="payload-pre-processing" containerID="cri-o://29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7" gracePeriod=25 Oct 30 03:30:53.650591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.650536 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5"] Oct 30 03:30:53.650843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.650779 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" podUID="2a1196db-cabd-47e3-a02d-4f2a64fc37a6" containerName="maas-api" containerID="cri-o://e3a52c7398346a49f6953fd1bbe6ac9a89f5f4340b2cff45fa08cba2a3ffea31" gracePeriod=30 Oct 30 03:30:53.715125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.715063 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:53.785569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.785541 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:30:53.795500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.795477 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:30:53.807099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.807066 4442 generic.go:356] "Generic (PLEG): container finished" podID="fbbdb317-95af-4141-8f1b-55d7eff61a27" containerID="29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7" exitCode=0 Oct 30 03:30:53.807099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.807111 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" Oct 30 03:30:53.807328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.807146 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" event={"ID":"fbbdb317-95af-4141-8f1b-55d7eff61a27","Type":"ContainerDied","Data":"29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7"} Oct 30 03:30:53.807328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.807251 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7" event={"ID":"fbbdb317-95af-4141-8f1b-55d7eff61a27","Type":"ContainerDied","Data":"1ec42bb5db2c25204fe69437d2d1af03444d561dabec4fa484be578d0b8e84f8"} Oct 30 03:30:53.807328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.807282 4442 scope.go:122] "RemoveContainer" containerID="29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7" Oct 30 03:30:53.808901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.808821 4442 generic.go:356] "Generic (PLEG): container finished" podID="dfa35cf8-883a-494f-b949-41cd36f73b41" containerID="12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7" exitCode=0 Oct 30 03:30:53.808901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.808871 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" Oct 30 03:30:53.808901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.808892 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" event={"ID":"dfa35cf8-883a-494f-b949-41cd36f73b41","Type":"ContainerDied","Data":"12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7"} Oct 30 03:30:53.809080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.808926 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr" event={"ID":"dfa35cf8-883a-494f-b949-41cd36f73b41","Type":"ContainerDied","Data":"f8a84ce2b4cc75dd2d957a7be3753bb8822f4a433ef0c1d4446868929c9ae4b4"} Oct 30 03:30:53.810687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.810666 4442 generic.go:356] "Generic (PLEG): container finished" podID="2a1196db-cabd-47e3-a02d-4f2a64fc37a6" containerID="e3a52c7398346a49f6953fd1bbe6ac9a89f5f4340b2cff45fa08cba2a3ffea31" exitCode=0 Oct 30 03:30:53.810815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.810745 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" event={"ID":"2a1196db-cabd-47e3-a02d-4f2a64fc37a6","Type":"ContainerDied","Data":"e3a52c7398346a49f6953fd1bbe6ac9a89f5f4340b2cff45fa08cba2a3ffea31"} Oct 30 03:30:53.817528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.817510 4442 scope.go:122] "RemoveContainer" containerID="29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7" Oct 30 03:30:53.817817 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:30:53.817797 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7\": container with ID starting with 29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7 not found: ID does not exist" containerID="29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7" Oct 30 03:30:53.817936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.817918 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7"} err="failed to get container status \"29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7\": rpc error: code = NotFound desc = could not find container \"29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7\": container with ID starting with 29c75a7a771827180d5c8e5b6e1e215cc2958fdb1cfa2b90ec39ff28d61ca8d7 not found: ID does not exist" Oct 30 03:30:53.818006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.817997 4442 scope.go:122] "RemoveContainer" containerID="12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7" Oct 30 03:30:53.822581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.822557 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbdb317-95af-4141-8f1b-55d7eff61a27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbdb317-95af-4141-8f1b-55d7eff61a27-plugins-config-volume\") pod \"fbbdb317-95af-4141-8f1b-55d7eff61a27\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " Oct 30 03:30:53.822742 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.822720 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa35cf8-883a-494f-b949-41cd36f73b41-kube-api-access-r9pvq\" (UniqueName: \"kubernetes.io/projected/dfa35cf8-883a-494f-b949-41cd36f73b41-kube-api-access-r9pvq\") pod \"dfa35cf8-883a-494f-b949-41cd36f73b41\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " Oct 30 03:30:53.823729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.823365 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa35cf8-883a-494f-b949-41cd36f73b41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa35cf8-883a-494f-b949-41cd36f73b41-plugins-config-volume\") pod \"dfa35cf8-883a-494f-b949-41cd36f73b41\" (UID: \"dfa35cf8-883a-494f-b949-41cd36f73b41\") " Oct 30 03:30:53.823729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.822971 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbdb317-95af-4141-8f1b-55d7eff61a27-plugins-config-volume" pod "fbbdb317-95af-4141-8f1b-55d7eff61a27" (UID: "fbbdb317-95af-4141-8f1b-55d7eff61a27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:53.823729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.823414 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbdb317-95af-4141-8f1b-55d7eff61a27-kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/fbbdb317-95af-4141-8f1b-55d7eff61a27-kube-api-access-l8884\") pod \"fbbdb317-95af-4141-8f1b-55d7eff61a27\" (UID: \"fbbdb317-95af-4141-8f1b-55d7eff61a27\") " Oct 30 03:30:53.823941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.823766 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa35cf8-883a-494f-b949-41cd36f73b41-plugins-config-volume" pod "dfa35cf8-883a-494f-b949-41cd36f73b41" (UID: "dfa35cf8-883a-494f-b949-41cd36f73b41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:53.824045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.823991 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfa35cf8-883a-494f-b949-41cd36f73b41-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:53.824045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.824015 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbbdb317-95af-4141-8f1b-55d7eff61a27-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:53.825660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.825633 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa35cf8-883a-494f-b949-41cd36f73b41-kube-api-access-r9pvq" pod "dfa35cf8-883a-494f-b949-41cd36f73b41" (UID: "dfa35cf8-883a-494f-b949-41cd36f73b41"). InnerVolumeSpecName "kube-api-access-r9pvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:53.830531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.830277 4442 scope.go:122] "RemoveContainer" containerID="12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7" Oct 30 03:30:53.830531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.830475 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbdb317-95af-4141-8f1b-55d7eff61a27-kube-api-access-l8884" pod "fbbdb317-95af-4141-8f1b-55d7eff61a27" (UID: "fbbdb317-95af-4141-8f1b-55d7eff61a27"). InnerVolumeSpecName "kube-api-access-l8884". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:53.830686 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:30:53.830628 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7\": container with ID starting with 12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7 not found: ID does not exist" containerID="12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7" Oct 30 03:30:53.830770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.830673 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7"} err="failed to get container status \"12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7\": rpc error: code = NotFound desc = could not find container \"12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7\": container with ID starting with 12f6a062126bc4dcf2e59dfc7e7bf773f40a850b03c5c18f605c8dd6f12f41d7 not found: ID does not exist" Oct 30 03:30:53.948226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.926250 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:30:53.948226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.927344 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9pvq\" (UniqueName: \"kubernetes.io/projected/dfa35cf8-883a-494f-b949-41cd36f73b41-kube-api-access-r9pvq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:53.948226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:53.927369 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8884\" (UniqueName: \"kubernetes.io/projected/fbbdb317-95af-4141-8f1b-55d7eff61a27-kube-api-access-l8884\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:54.028091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.028064 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-openshift-service-ca\") pod \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " Oct 30 03:30:54.028286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.028138 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-metrics-certs\") pod \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " Oct 30 03:30:54.028286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.028214 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls\") pod \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " Oct 30 03:30:54.028286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.028257 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-kube-api-access-t2pxn\" (UniqueName: \"kubernetes.io/projected/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-kube-api-access-t2pxn\") pod \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\" (UID: \"2a1196db-cabd-47e3-a02d-4f2a64fc37a6\") " Oct 30 03:30:54.028492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.028453 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-openshift-service-ca" pod "2a1196db-cabd-47e3-a02d-4f2a64fc37a6" (UID: "2a1196db-cabd-47e3-a02d-4f2a64fc37a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:30:54.028621 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.028600 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:54.030412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.030385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-metrics-certs" pod "2a1196db-cabd-47e3-a02d-4f2a64fc37a6" (UID: "2a1196db-cabd-47e3-a02d-4f2a64fc37a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:54.030412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.030385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-kube-api-access-t2pxn" pod "2a1196db-cabd-47e3-a02d-4f2a64fc37a6" (UID: "2a1196db-cabd-47e3-a02d-4f2a64fc37a6"). InnerVolumeSpecName "kube-api-access-t2pxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:30:54.030546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.030449 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls" pod "2a1196db-cabd-47e3-a02d-4f2a64fc37a6" (UID: "2a1196db-cabd-47e3-a02d-4f2a64fc37a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:30:54.158812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.141288 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2pxn\" (UniqueName: \"kubernetes.io/projected/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-kube-api-access-t2pxn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:54.158812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.141326 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:54.158812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.141342 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a1196db-cabd-47e3-a02d-4f2a64fc37a6-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:30:54.227842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.227815 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr"] Oct 30 03:30:54.238292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.238266 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-dc3196-77444cb88b-c28pr"] Oct 30 03:30:54.260532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.260511 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7"] Oct 30 03:30:54.265843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.265816 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-dc3196-6778b444ff-mgsg7"] Oct 30 03:30:54.398839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.398804 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:54.724733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.724702 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:54.825805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.825777 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:54.830276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.830252 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" event={"ID":"2a1196db-cabd-47e3-a02d-4f2a64fc37a6","Type":"ContainerDied","Data":"337e2fef2f66e68896b750b88d906fd19a106e17dd3bdd12280a1562ab525070"} Oct 30 03:30:54.830416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.830302 4442 scope.go:122] "RemoveContainer" containerID="e3a52c7398346a49f6953fd1bbe6ac9a89f5f4340b2cff45fa08cba2a3ffea31" Oct 30 03:30:54.830416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.830261 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5" Oct 30 03:30:54.860944 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:30:54.860914 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:30:54.876656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.876633 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5"] Oct 30 03:30:54.887294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:54.887270 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-dc3196-6cc7995d84-t6kx5"] Oct 30 03:30:55.438792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:55.438723 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:55.714365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:55.714275 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:55.815333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:55.815292 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:55.824697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:55.824660 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm"] Oct 30 03:30:55.824947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:55.824918 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" containerID="cri-o://07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575" gracePeriod=30 Oct 30 03:30:55.901240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:55.901206 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:56.110144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:56.110111 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1196db-cabd-47e3-a02d-4f2a64fc37a6" path="/var/lib/kubelet/pods/2a1196db-cabd-47e3-a02d-4f2a64fc37a6/volumes" Oct 30 03:30:56.110580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:56.110562 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa35cf8-883a-494f-b949-41cd36f73b41" path="/var/lib/kubelet/pods/dfa35cf8-883a-494f-b949-41cd36f73b41/volumes" Oct 30 03:30:56.111029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:56.111016 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbdb317-95af-4141-8f1b-55d7eff61a27" path="/var/lib/kubelet/pods/fbbdb317-95af-4141-8f1b-55d7eff61a27/volumes" Oct 30 03:30:56.125336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:56.125310 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:57.819801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:57.819711 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:58.062187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:58.062156 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:30:59.086230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:59.086196 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd"] Oct 30 03:30:59.120131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:59.120100 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6"] Oct 30 03:30:59.239094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:59.239065 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:59.359233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:59.359144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:30:59.657888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:30:59.657792 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:00.149116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:00.149081 4442 patch_prober.go:29] interesting pod/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:31:00.149483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:00.149143 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:31:01.095996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.095930 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:31:01.133616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133588 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-data\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.133798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133646 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-istio-token\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-istio-token\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.133798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133698 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-kube-api-access-v46kc\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-kube-api-access-v46kc\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.133798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133742 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8235830-dbab-4486-8f1e-3689feb8544e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8235830-dbab-4486-8f1e-3689feb8544e-istiod-ca-cert\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.133798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133794 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-socket\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.134044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133865 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-envoy\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.134044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133887 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-data" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:31:01.134044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.133979 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-credential-socket\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.134044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134009 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8235830-dbab-4486-8f1e-3689feb8544e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8235830-dbab-4486-8f1e-3689feb8544e-istio-podinfo\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.134044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134036 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-certs\") pod \"f8235830-dbab-4486-8f1e-3689feb8544e\" (UID: \"f8235830-dbab-4486-8f1e-3689feb8544e\") " Oct 30 03:31:01.134328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134086 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-socket" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:31:01.134328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134178 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8235830-dbab-4486-8f1e-3689feb8544e-istiod-ca-cert" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:31:01.134328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134316 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-credential-socket" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:31:01.134648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134497 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-certs" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:31:01.134648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134638 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8235830-dbab-4486-8f1e-3689feb8544e-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.134775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134656 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.134775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134669 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.134775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134684 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.134775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.134697 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.136156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.136131 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-kube-api-access-v46kc" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "kube-api-access-v46kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:31:01.136284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.136242 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-envoy" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:31:01.136643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.136609 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-istio-token" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:31:01.136712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.136683 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8235830-dbab-4486-8f1e-3689feb8544e-istio-podinfo" pod "f8235830-dbab-4486-8f1e-3689feb8544e" (UID: "f8235830-dbab-4486-8f1e-3689feb8544e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:31:01.236375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.236343 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.236375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.236368 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v46kc\" (UniqueName: \"kubernetes.io/projected/f8235830-dbab-4486-8f1e-3689feb8544e-kube-api-access-v46kc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.236375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.236381 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8235830-dbab-4486-8f1e-3689feb8544e-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.236859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.236390 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8235830-dbab-4486-8f1e-3689feb8544e-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:01.419409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.419326 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:01.879731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.879697 4442 generic.go:356] "Generic (PLEG): container finished" podID="f8235830-dbab-4486-8f1e-3689feb8544e" containerID="07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575" exitCode=0 Oct 30 03:31:01.879952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.879785 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" Oct 30 03:31:01.879952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.879783 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" event={"ID":"f8235830-dbab-4486-8f1e-3689feb8544e","Type":"ContainerDied","Data":"07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575"} Oct 30 03:31:01.879952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.879878 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm" event={"ID":"f8235830-dbab-4486-8f1e-3689feb8544e","Type":"ContainerDied","Data":"0219853da41c63d1c910862acf20847963c748f14f9a0ba5f139e07dccac5484"} Oct 30 03:31:01.879952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.879908 4442 scope.go:122] "RemoveContainer" containerID="07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575" Oct 30 03:31:01.890096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.890081 4442 scope.go:122] "RemoveContainer" containerID="741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99" Oct 30 03:31:01.898860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.898840 4442 scope.go:122] "RemoveContainer" containerID="07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575" Oct 30 03:31:01.899138 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:31:01.899116 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575\": container with ID starting with 07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575 not found: ID does not exist" containerID="07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575" Oct 30 03:31:01.899183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.899148 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575"} err="failed to get container status \"07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575\": rpc error: code = NotFound desc = could not find container \"07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575\": container with ID starting with 07d75a9dce1f1e2f4e304294b333a8ff0e783d9ada10e6fe33d83738d5d1e575 not found: ID does not exist" Oct 30 03:31:01.899183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.899168 4442 scope.go:122] "RemoveContainer" containerID="741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99" Oct 30 03:31:01.899411 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:31:01.899396 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99\": container with ID starting with 741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99 not found: ID does not exist" containerID="741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99" Oct 30 03:31:01.899447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.899415 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99"} err="failed to get container status \"741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99\": rpc error: code = NotFound desc = could not find container \"741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99\": container with ID starting with 741ed3c51a1fb971ff68c4301f6f1eee2a728aaf5f11b033eb7eef3eda545c99 not found: ID does not exist" Oct 30 03:31:01.939862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.939827 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm"] Oct 30 03:31:01.951061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:01.951031 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-dc3196-openshift-default-5d5df65f49-j2cdm"] Oct 30 03:31:02.106922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:02.106889 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" path="/var/lib/kubelet/pods/f8235830-dbab-4486-8f1e-3689feb8544e/volumes" Oct 30 03:31:03.274996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:03.274966 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:04.086524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.086484 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" podUID="1422630a-fd28-4209-9316-586b74125bce" containerName="payload-processing" containerID="cri-o://22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f" gracePeriod=25 Oct 30 03:31:04.120871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.120828 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" podUID="a384c723-3eb5-431c-9955-255cf34fdb05" containerName="payload-pre-processing" containerID="cri-o://bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408" gracePeriod=25 Oct 30 03:31:04.384627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.384600 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:31:04.417402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.417378 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:31:04.471825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.471796 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a384c723-3eb5-431c-9955-255cf34fdb05-kube-api-access-qvf6f\" (UniqueName: \"kubernetes.io/projected/a384c723-3eb5-431c-9955-255cf34fdb05-kube-api-access-qvf6f\") pod \"a384c723-3eb5-431c-9955-255cf34fdb05\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " Oct 30 03:31:04.472005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.471842 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1422630a-fd28-4209-9316-586b74125bce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1422630a-fd28-4209-9316-586b74125bce-plugins-config-volume\") pod \"1422630a-fd28-4209-9316-586b74125bce\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " Oct 30 03:31:04.472005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.471950 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1422630a-fd28-4209-9316-586b74125bce-kube-api-access-69h4x\" (UniqueName: \"kubernetes.io/projected/1422630a-fd28-4209-9316-586b74125bce-kube-api-access-69h4x\") pod \"1422630a-fd28-4209-9316-586b74125bce\" (UID: \"1422630a-fd28-4209-9316-586b74125bce\") " Oct 30 03:31:04.472135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.472119 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a384c723-3eb5-431c-9955-255cf34fdb05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a384c723-3eb5-431c-9955-255cf34fdb05-plugins-config-volume\") pod \"a384c723-3eb5-431c-9955-255cf34fdb05\" (UID: \"a384c723-3eb5-431c-9955-255cf34fdb05\") " Oct 30 03:31:04.472844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.472802 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a384c723-3eb5-431c-9955-255cf34fdb05-plugins-config-volume" pod "a384c723-3eb5-431c-9955-255cf34fdb05" (UID: "a384c723-3eb5-431c-9955-255cf34fdb05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:31:04.472844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.472808 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1422630a-fd28-4209-9316-586b74125bce-plugins-config-volume" pod "1422630a-fd28-4209-9316-586b74125bce" (UID: "1422630a-fd28-4209-9316-586b74125bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:31:04.474831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.474742 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1422630a-fd28-4209-9316-586b74125bce-kube-api-access-69h4x" pod "1422630a-fd28-4209-9316-586b74125bce" (UID: "1422630a-fd28-4209-9316-586b74125bce"). InnerVolumeSpecName "kube-api-access-69h4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:31:04.475416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.475383 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a384c723-3eb5-431c-9955-255cf34fdb05-kube-api-access-qvf6f" pod "a384c723-3eb5-431c-9955-255cf34fdb05" (UID: "a384c723-3eb5-431c-9955-255cf34fdb05"). InnerVolumeSpecName "kube-api-access-qvf6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:31:04.500944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.500915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:04.573060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.573036 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69h4x\" (UniqueName: \"kubernetes.io/projected/1422630a-fd28-4209-9316-586b74125bce-kube-api-access-69h4x\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:04.573060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.573058 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a384c723-3eb5-431c-9955-255cf34fdb05-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:04.573271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.573067 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvf6f\" (UniqueName: \"kubernetes.io/projected/a384c723-3eb5-431c-9955-255cf34fdb05-kube-api-access-qvf6f\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:04.573271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.573078 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1422630a-fd28-4209-9316-586b74125bce-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:31:04.931779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.928984 4442 generic.go:356] "Generic (PLEG): container finished" podID="1422630a-fd28-4209-9316-586b74125bce" containerID="22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f" exitCode=0 Oct 30 03:31:04.931779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.929131 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" event={"ID":"1422630a-fd28-4209-9316-586b74125bce","Type":"ContainerDied","Data":"22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f"} Oct 30 03:31:04.931779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.929186 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" event={"ID":"1422630a-fd28-4209-9316-586b74125bce","Type":"ContainerDied","Data":"66232c5a1aa5d8588c084116dc6a01bab70443ae445bbbfed1e26d430d690f55"} Oct 30 03:31:04.931779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.929215 4442 scope.go:122] "RemoveContainer" containerID="22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f" Oct 30 03:31:04.931779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.929320 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" Oct 30 03:31:04.963685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.958510 4442 generic.go:356] "Generic (PLEG): container finished" podID="a384c723-3eb5-431c-9955-255cf34fdb05" containerID="bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408" exitCode=0 Oct 30 03:31:04.963685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.958619 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" event={"ID":"a384c723-3eb5-431c-9955-255cf34fdb05","Type":"ContainerDied","Data":"bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408"} Oct 30 03:31:04.963685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.958672 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" event={"ID":"a384c723-3eb5-431c-9955-255cf34fdb05","Type":"ContainerDied","Data":"0574b374a7f70ec7565c1064f57fcdd8bd0705bbd6124892b79705704726fa31"} Oct 30 03:31:04.966852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.966829 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" Oct 30 03:31:04.976699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.975979 4442 scope.go:122] "RemoveContainer" containerID="22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f" Oct 30 03:31:04.976699 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:31:04.976323 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f\": container with ID starting with 22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f not found: ID does not exist" containerID="22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f" Oct 30 03:31:04.976699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.976356 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f"} err="failed to get container status \"22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f\": rpc error: code = NotFound desc = could not find container \"22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f\": container with ID starting with 22b983789f67ba3f66c64cbdd7de219a2838cb22093ad2ac0986011bbafc549f not found: ID does not exist" Oct 30 03:31:04.976699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.976382 4442 scope.go:122] "RemoveContainer" containerID="bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408" Oct 30 03:31:04.989672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.989645 4442 scope.go:122] "RemoveContainer" containerID="bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408" Oct 30 03:31:04.990133 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:31:04.990112 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408\": container with ID starting with bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408 not found: ID does not exist" containerID="bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408" Oct 30 03:31:04.990229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:04.990141 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408"} err="failed to get container status \"bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408\": rpc error: code = NotFound desc = could not find container \"bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408\": container with ID starting with bdfe74695b54f82a647be66d00dcd0e3c977fde81f8221cd850f26ccb15a9408 not found: ID does not exist" Oct 30 03:31:05.017389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.017367 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd"] Oct 30 03:31:05.028202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.028180 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd"] Oct 30 03:31:05.059171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.059144 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6"] Oct 30 03:31:05.062174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.062153 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6"] Oct 30 03:31:05.121285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.121262 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:05.223675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.223602 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:05.355518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.355481 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-773fbb-65884f8944-njspd" podUID="1422630a-fd28-4209-9316-586b74125bce" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.119:50052\" within 1s: context deadline exceeded" Oct 30 03:31:05.355691 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:31:05.355564 4442 logging.go:55] [core] [Channel #1353 SubChannel #1354]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.118:50052", ServerName: "10.134.0.118:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.118:50052: operation was canceled" Oct 30 03:31:05.355691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.355491 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-773fbb-7f486964c-cpcl6" podUID="a384c723-3eb5-431c-9955-255cf34fdb05" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.118:50052\" within 1s: context deadline exceeded" Oct 30 03:31:05.355691 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:31:05.355569 4442 logging.go:55] [core] [Channel #1352 SubChannel #1355]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.119:50052", ServerName: "10.134.0.119:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.119:50052: operation was canceled" Oct 30 03:31:05.831023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:05.830986 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:06.109633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:06.109545 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1422630a-fd28-4209-9316-586b74125bce" path="/var/lib/kubelet/pods/1422630a-fd28-4209-9316-586b74125bce/volumes" Oct 30 03:31:06.110028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:06.110013 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a384c723-3eb5-431c-9955-255cf34fdb05" path="/var/lib/kubelet/pods/a384c723-3eb5-431c-9955-255cf34fdb05/volumes" Oct 30 03:31:06.313039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:06.312992 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:07.039512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:07.039480 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:08.626789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:08.626744 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:10.315496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:10.315465 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:11.901550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:11.901515 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:13.502400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:13.502368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:15.188307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:15.188273 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:16.908427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:16.908393 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:17.813426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:17.813391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:18.627941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:18.627910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:20.280464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:20.280432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:22.033775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:22.033722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:23.046212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:23.046178 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:31:23.897934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:23.897846 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:24.859654 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:31:24.859619 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:31:25.720430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:25.720386 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:27.500677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:27.500628 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:29.327112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:29.327081 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:31.156674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:31.156634 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:33.016220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:33.016190 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:34.806841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:34.806814 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:36.474875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:36.474842 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:38.306366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:38.306331 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:40.132811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:40.132767 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:41.813926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:41.813892 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:43.507094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:43.507063 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:44.647905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:44.647847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:44.749043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:44.749003 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:45.321973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:45.321938 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:46.714797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:46.714719 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:47.047345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:47.047304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:47.746533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:47.746504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:48.760852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:48.760823 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:49.717683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:49.717637 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:50.395598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:50.395513 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:52.135396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:52.135343 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:53.846400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:53.846363 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:54.859985 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:31:54.859949 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:31:55.627745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:55.627717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:57.425016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:57.424977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:31:59.235093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:31:59.235055 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:01.059905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:01.059859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:02.713989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:02.713924 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:04.438195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:04.438137 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:06.111658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:06.111620 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:07.803347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:07.803310 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:09.419262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:09.419173 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:11.209936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:11.209897 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:12.915066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:12.915034 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:14.687209 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:14.687181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:16.489086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:16.489010 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:18.217061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:18.217009 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:19.917596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:19.917541 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:21.645735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:21.645682 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:23.401123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:23.401084 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:24.860067 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:32:24.860027 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:32:25.187573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:25.187482 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:26.898361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:26.898328 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:28.630654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:28.630623 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:30.307308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:30.307268 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:32.018798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:32.018736 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:33.752039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:33.752000 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:35.481683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:35.481652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:37.138984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:37.138948 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:38.918915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:38.918855 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:40.625704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:40.625674 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:42.316189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:42.316155 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:44.007234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:44.007198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:45.713616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:45.713584 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:46.395996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:46.395962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:46.821983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:46.821953 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:47.405320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:47.405290 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:47.861286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:47.861257 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:49.204689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:49.204651 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:49.832387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:49.832357 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:50.835297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:50.835261 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:50.930781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:50.913446 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr"] Oct 30 03:32:50.931308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:50.931267 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" containerID="cri-o://53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace" gracePeriod=30 Oct 30 03:32:52.520982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:52.520950 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:53.980678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:53.980643 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:54.064290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:54.064265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:54.169332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:54.169301 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:54.860848 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:32:54.860799 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:32:55.186320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:55.186220 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:55.287438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:55.287406 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:55.991640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:55.991616 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:56.211338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.211315 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:32:56.353202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353171 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-data\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353223 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-kube-api-access-vsf8q\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-kube-api-access-vsf8q\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353260 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-podinfo\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353307 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-socket\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353349 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-envoy\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353380 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istiod-ca-cert\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353441 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-token\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-token\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353468 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-data" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:56.353672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353504 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-certs\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353531 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-credential-socket\") pod \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\" (UID: \"10f928cd-d051-43f2-a6d2-6d40bdc0c635\") " Oct 30 03:32:56.353922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353811 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-certs" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:56.353980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.353914 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-socket" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:56.354158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.354122 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istiod-ca-cert" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:32:56.354406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.354379 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-credential-socket" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:56.355007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.354984 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.355089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.355014 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.355089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.355030 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.355089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.355043 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.355089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.355055 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.356120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.356096 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-podinfo" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:32:56.356120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.356109 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-envoy" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:32:56.356292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.356131 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-kube-api-access-vsf8q" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "kube-api-access-vsf8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:32:56.357839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.357819 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-token" pod "10f928cd-d051-43f2-a6d2-6d40bdc0c635" (UID: "10f928cd-d051-43f2-a6d2-6d40bdc0c635"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:32:56.375969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.375949 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:32:56.385541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.385518 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:32:56.456193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.456123 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsf8q\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-kube-api-access-vsf8q\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.456193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.456148 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.456193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.456160 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.456193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.456167 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f928cd-d051-43f2-a6d2-6d40bdc0c635-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:32:56.562932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.562900 4442 generic.go:356] "Generic (PLEG): container finished" podID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerID="53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace" exitCode=0 Oct 30 03:32:56.563079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.562950 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" Oct 30 03:32:56.563079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.562988 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" event={"ID":"10f928cd-d051-43f2-a6d2-6d40bdc0c635","Type":"ContainerDied","Data":"53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace"} Oct 30 03:32:56.563079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.563044 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr" event={"ID":"10f928cd-d051-43f2-a6d2-6d40bdc0c635","Type":"ContainerDied","Data":"d18de72a658aa7c23e24e4b9d75dfdb778ff8bc9a624b2051eb8f2adfd246eec"} Oct 30 03:32:56.563079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.563072 4442 scope.go:122] "RemoveContainer" containerID="53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace" Oct 30 03:32:56.572875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.572859 4442 scope.go:122] "RemoveContainer" containerID="53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace" Oct 30 03:32:56.573132 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:32:56.573112 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace\": container with ID starting with 53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace not found: ID does not exist" containerID="53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace" Oct 30 03:32:56.573207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.573143 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace"} err="failed to get container status \"53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace\": rpc error: code = NotFound desc = could not find container \"53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace\": container with ID starting with 53ee44b40b0542e1fc8872279da2ab0c2c24bc030b3c173936a054688b3d3ace not found: ID does not exist" Oct 30 03:32:56.629597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.629570 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr"] Oct 30 03:32:56.641841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.641818 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-k6nmr"] Oct 30 03:32:56.745000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:56.744919 4442 scope.go:122] "RemoveContainer" containerID="fa67d0908a98f9c7cdb56bbede5bddb7a73a3adb4441b388583e9c7cb4a28f68" Oct 30 03:32:57.537977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:57.537942 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:57.919931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:57.919844 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:58.024318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:58.024283 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:32:58.106135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:58.106093 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" path="/var/lib/kubelet/pods/10f928cd-d051-43f2-a6d2-6d40bdc0c635/volumes" Oct 30 03:32:59.185980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:32:59.185949 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:00.907534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:00.907503 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:02.629410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:02.629376 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:04.287424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:04.287394 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:06.026698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:06.026669 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:07.712925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:07.712882 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:09.424006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:09.423972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:11.124926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:11.124891 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:12.926459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:12.926418 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:14.619545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:14.619510 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:16.404700 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:16.404662 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:18.118795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:18.118727 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:19.788233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:19.788200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:21.449091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:21.449059 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:23.113729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:23.113697 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:24.859807 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:33:24.859743 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:33:24.913280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:24.913237 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:26.719883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:26.719847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:28.407245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:28.407207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:30.127875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:30.127848 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:31.843548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:31.841132 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:33.495738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:33.495701 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:35.261044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:35.261006 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:36.989154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:36.989120 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:38.706408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:38.706328 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:40.428392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:40.428307 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:42.133014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:42.132979 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:43.941834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:43.941799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:45.623960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:45.623921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:46.934909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:46.934875 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:47.329881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:47.329837 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:47.979115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:47.979085 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:49.026873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:49.026839 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:49.959276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:49.959244 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:50.708285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:50.708251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:51.078770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:51.078726 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s"] Oct 30 03:33:51.078981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:51.078957 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" containerID="cri-o://2e664f97e6322208d7d1b808225ee821ff22f368b03ddc65926e257079df58ad" gracePeriod=30 Oct 30 03:33:52.416269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:52.416232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:54.159514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:54.159463 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:54.860209 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:33:54.860176 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:33:55.719413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:55.719381 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:56.342361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.342336 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:33:56.469946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.469851 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-envoy\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.469946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.469906 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea287de4-a030-4e5e-b106-451729ab7806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea287de4-a030-4e5e-b106-451729ab7806-istiod-ca-cert\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.469975 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea287de4-a030-4e5e-b106-451729ab7806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea287de4-a030-4e5e-b106-451729ab7806-istio-podinfo\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470001 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-credential-socket\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470031 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-istio-token\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-istio-token\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470133 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-data\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470165 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-socket\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470204 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-certs\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470245 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-kube-api-access-6c468\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-kube-api-access-6c468\") pod \"ea287de4-a030-4e5e-b106-451729ab7806\" (UID: \"ea287de4-a030-4e5e-b106-451729ab7806\") " Oct 30 03:33:56.470455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470368 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-credential-socket" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:56.470816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470600 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-data" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:56.470816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470662 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.470816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470659 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea287de4-a030-4e5e-b106-451729ab7806-istiod-ca-cert" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:33:56.470994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470919 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-certs" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:56.470994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.470947 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-socket" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:56.472324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.472295 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-envoy" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:33:56.472699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.472681 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea287de4-a030-4e5e-b106-451729ab7806-istio-podinfo" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:33:56.472869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.472852 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-istio-token" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:33:56.472944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.472925 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-kube-api-access-6c468" pod "ea287de4-a030-4e5e-b106-451729ab7806" (UID: "ea287de4-a030-4e5e-b106-451729ab7806"). InnerVolumeSpecName "kube-api-access-6c468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:33:56.571398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571369 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.571398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571395 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea287de4-a030-4e5e-b106-451729ab7806-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.571398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571404 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea287de4-a030-4e5e-b106-451729ab7806-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.571643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571412 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.571643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571420 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.571643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571427 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.571643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571434 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea287de4-a030-4e5e-b106-451729ab7806-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.571643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.571444 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c468\" (UniqueName: \"kubernetes.io/projected/ea287de4-a030-4e5e-b106-451729ab7806-kube-api-access-6c468\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:33:56.780728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.780699 4442 scope.go:122] "RemoveContainer" containerID="2e664f97e6322208d7d1b808225ee821ff22f368b03ddc65926e257079df58ad" Oct 30 03:33:56.875522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.875483 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" event={"ID":"ea287de4-a030-4e5e-b106-451729ab7806","Type":"ContainerDied","Data":"2e664f97e6322208d7d1b808225ee821ff22f368b03ddc65926e257079df58ad"} Oct 30 03:33:56.875522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.875506 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" Oct 30 03:33:56.875741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.875544 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s" event={"ID":"ea287de4-a030-4e5e-b106-451729ab7806","Type":"ContainerDied","Data":"56e0ba5475a5c057ae1b2f4cd9ef03a09327cea7cef22964cfa85a114f85a8cc"} Oct 30 03:33:56.958312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.958269 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s"] Oct 30 03:33:56.961046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:56.961024 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-h6l2s"] Oct 30 03:33:57.317832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:57.317800 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:33:58.107089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:58.107060 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea287de4-a030-4e5e-b106-451729ab7806" path="/var/lib/kubelet/pods/ea287de4-a030-4e5e-b106-451729ab7806/volumes" Oct 30 03:33:59.003576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:33:59.003536 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:00.704112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:00.704068 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:02.295461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:02.295427 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:03.903407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:03.903369 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:05.501799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:05.501769 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:07.098526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:07.098490 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:08.800559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:08.800526 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:10.379409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:10.379373 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:11.975225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:11.975193 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:13.564440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:13.564404 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:15.235934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:15.235907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:16.836990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:16.836957 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:18.528996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:18.528967 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:20.236481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:20.236451 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:21.910505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:21.910467 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:23.500540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:23.500506 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:24.860123 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:34:24.860088 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:34:25.226383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:25.226304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:26.901338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:26.901304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:28.601696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:28.601665 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:28.743863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:28.743831 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l"] Oct 30 03:34:28.744069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:28.744027 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" containerID="cri-o://b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c" gracePeriod=30 Oct 30 03:34:30.314443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:30.314406 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:32.028115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:32.028081 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:32.121399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:32.121364 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6"] Oct 30 03:34:32.121634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:32.121610 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" containerID="cri-o://c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898" gracePeriod=30 Oct 30 03:34:32.132923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:32.132895 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn"] Oct 30 03:34:32.133104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:32.133082 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" containerID="cri-o://8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c" gracePeriod=30 Oct 30 03:34:33.732733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:33.732695 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:33.997695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:33.997645 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:34:34.069816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.069787 4442 generic.go:356] "Generic (PLEG): container finished" podID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerID="b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c" exitCode=0 Oct 30 03:34:34.070060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.069835 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" event={"ID":"ee8ad860-7156-451c-9b66-001c8bbd7107","Type":"ContainerDied","Data":"b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c"} Oct 30 03:34:34.070205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.070099 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" event={"ID":"ee8ad860-7156-451c-9b66-001c8bbd7107","Type":"ContainerDied","Data":"17af3f461c2594eb0e970b7b85672c4bdcce3f75b28a704453e3801b4861c074"} Oct 30 03:34:34.070205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.069865 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l" Oct 30 03:34:34.070205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.070134 4442 scope.go:122] "RemoveContainer" containerID="b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c" Oct 30 03:34:34.079391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.079371 4442 scope.go:122] "RemoveContainer" containerID="b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c" Oct 30 03:34:34.079668 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:34:34.079649 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c\": container with ID starting with b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c not found: ID does not exist" containerID="b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c" Oct 30 03:34:34.079784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.079675 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c"} err="failed to get container status \"b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c\": rpc error: code = NotFound desc = could not find container \"b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c\": container with ID starting with b94f65b72d719954cd469cd7430ee4efaf645fa2c7835973a775416cdf9a867c not found: ID does not exist" Oct 30 03:34:34.098770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.098723 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-socket\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.098887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.098823 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-credential-socket\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.098887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.098860 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-certs\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.098988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.098934 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-podinfo\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.098988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.098982 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-token\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-token\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.099092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099046 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-envoy\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.099092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099042 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-socket" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:34.099194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099109 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee8ad860-7156-451c-9b66-001c8bbd7107-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ad860-7156-451c-9b66-001c8bbd7107-istiod-ca-cert\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.099194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099159 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-kube-api-access-mwl92\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-kube-api-access-mwl92\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.099304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099229 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-certs" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:34.099304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099256 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-data\") pod \"ee8ad860-7156-451c-9b66-001c8bbd7107\" (UID: \"ee8ad860-7156-451c-9b66-001c8bbd7107\") " Oct 30 03:34:34.099414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099342 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-credential-socket" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:34.099670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099638 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.099670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099662 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.101809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.099676 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.101809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.100541 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-data" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:34.101809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.101222 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8ad860-7156-451c-9b66-001c8bbd7107-istiod-ca-cert" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:34:34.103612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.103574 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-envoy" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:34.103724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.103693 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-kube-api-access-mwl92" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "kube-api-access-mwl92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:34.104108 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.104084 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-token" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:34.104204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.104147 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-podinfo" pod "ee8ad860-7156-451c-9b66-001c8bbd7107" (UID: "ee8ad860-7156-451c-9b66-001c8bbd7107"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:34:34.200729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.200698 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.200729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.200732 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8ad860-7156-451c-9b66-001c8bbd7107-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.200951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.200748 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwl92\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-kube-api-access-mwl92\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.200951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.200799 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.200951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.200813 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.200951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.200824 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8ad860-7156-451c-9b66-001c8bbd7107-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:34.405986 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.405933 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l"] Oct 30 03:34:34.416890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:34.416855 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-x4g8l"] Oct 30 03:34:35.356867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:35.356823 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:36.106089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:36.106055 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" path="/var/lib/kubelet/pods/ee8ad860-7156-451c-9b66-001c8bbd7107/volumes" Oct 30 03:34:36.719325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:36.719295 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:37.049581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.049547 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:37.323151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.323057 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:37.398124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.398095 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:34:37.421971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.421946 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:34:37.546327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546297 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-istio-token\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-istio-token\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546336 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-socket\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546357 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10abd507-0528-408f-adc7-cdc1fa787d54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abd507-0528-408f-adc7-cdc1fa787d54-istiod-ca-cert\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546412 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istiod-ca-cert\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.546513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546445 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-envoy\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.546513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546471 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-token\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-token\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.546513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546509 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-envoy\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546564 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-podinfo\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546605 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-kube-api-access-lcmlj\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-kube-api-access-lcmlj\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546644 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-certs\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546673 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-socket\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546712 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-kube-api-access-kv447\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-kube-api-access-kv447\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546718 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-socket" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546738 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10abd507-0528-408f-adc7-cdc1fa787d54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abd507-0528-408f-adc7-cdc1fa787d54-istio-podinfo\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546791 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-data\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546818 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-credential-socket\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546831 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10abd507-0528-408f-adc7-cdc1fa787d54-istiod-ca-cert" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:34:37.546870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546862 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-certs\") pod \"10abd507-0528-408f-adc7-cdc1fa787d54\" (UID: \"10abd507-0528-408f-adc7-cdc1fa787d54\") " Oct 30 03:34:37.547434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546907 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-credential-socket\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.547434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.546949 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-data\") pod \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\" (UID: \"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0\") " Oct 30 03:34:37.547434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547192 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-data" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.547434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547214 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istiod-ca-cert" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:34:37.547434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547388 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-credential-socket" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.547645 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547566 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-certs" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.547902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547868 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.547902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547891 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.548078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547903 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.548078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547957 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.548078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547972 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.548078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547984 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10abd507-0528-408f-adc7-cdc1fa787d54-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.548078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.547995 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-certs" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.548300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.548076 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-socket" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.548300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.548161 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-credential-socket" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.548363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.548311 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-data" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.548690 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.548660 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:37.549412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.549389 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-istio-token" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:37.550074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.550053 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-kube-api-access-kv447" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "kube-api-access-kv447". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:37.550423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.550392 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-kube-api-access-lcmlj" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "kube-api-access-lcmlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:37.550423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.550413 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-envoy" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.550592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.550579 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-envoy" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:37.550940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.550910 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-token" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:37.551096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.551077 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-podinfo" pod "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" (UID: "02f9e7b0-9769-4a43-b2b5-8ab40addd8c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:34:37.551149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.551118 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10abd507-0528-408f-adc7-cdc1fa787d54-istio-podinfo" pod "10abd507-0528-408f-adc7-cdc1fa787d54" (UID: "10abd507-0528-408f-adc7-cdc1fa787d54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:34:37.648963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.648874 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.649368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.649348 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.649613 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.649599 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10abd507-0528-408f-adc7-cdc1fa787d54-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.649737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.649721 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.649867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.649745 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcmlj\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-kube-api-access-lcmlj\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.649867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.649776 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.650066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.650048 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.650161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.650072 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv447\" (UniqueName: \"kubernetes.io/projected/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-kube-api-access-kv447\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.650227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.650117 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10abd507-0528-408f-adc7-cdc1fa787d54-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.650227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.650175 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.650227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.650189 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.650227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.650202 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10abd507-0528-408f-adc7-cdc1fa787d54-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:37.660695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.660662 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:37.922517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:37.922407 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:38.091449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.091415 4442 generic.go:356] "Generic (PLEG): container finished" podID="10abd507-0528-408f-adc7-cdc1fa787d54" containerID="8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c" exitCode=0 Oct 30 03:34:38.091654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.091466 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" Oct 30 03:34:38.091654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.091498 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" event={"ID":"10abd507-0528-408f-adc7-cdc1fa787d54","Type":"ContainerDied","Data":"8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c"} Oct 30 03:34:38.091654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.091551 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn" event={"ID":"10abd507-0528-408f-adc7-cdc1fa787d54","Type":"ContainerDied","Data":"19779af1f3733f1adce6cf864b8a9f070a6b59becd03c35b9f5adec2fc0b7940"} Oct 30 03:34:38.091654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.091573 4442 scope.go:122] "RemoveContainer" containerID="8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c" Oct 30 03:34:38.093387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.093349 4442 generic.go:356] "Generic (PLEG): container finished" podID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerID="c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898" exitCode=0 Oct 30 03:34:38.093511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.093445 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" Oct 30 03:34:38.093570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.093537 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" event={"ID":"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0","Type":"ContainerDied","Data":"c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898"} Oct 30 03:34:38.093631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.093592 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6" event={"ID":"02f9e7b0-9769-4a43-b2b5-8ab40addd8c0","Type":"ContainerDied","Data":"9f2d0cc060cd43bdae07d38d06c029424369cd83b554cf95d3288f090572d54e"} Oct 30 03:34:38.101939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.101920 4442 scope.go:122] "RemoveContainer" containerID="8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c" Oct 30 03:34:38.102369 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:34:38.102235 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c\": container with ID starting with 8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c not found: ID does not exist" containerID="8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c" Oct 30 03:34:38.102369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.102263 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c"} err="failed to get container status \"8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c\": rpc error: code = NotFound desc = could not find container \"8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c\": container with ID starting with 8080c904d8a2c89890c2890fc527937f70f8b906b1f398d9e03e9a109d956b3c not found: ID does not exist" Oct 30 03:34:38.102369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.102280 4442 scope.go:122] "RemoveContainer" containerID="c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898" Oct 30 03:34:38.105797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.105770 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:38.111380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.111365 4442 scope.go:122] "RemoveContainer" containerID="c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898" Oct 30 03:34:38.111664 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:34:38.111643 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898\": container with ID starting with c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898 not found: ID does not exist" containerID="c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898" Oct 30 03:34:38.111728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.111674 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898"} err="failed to get container status \"c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898\": rpc error: code = NotFound desc = could not find container \"c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898\": container with ID starting with c8f88b7c0d2382c305c092c4d594ef460010638b888b116f4e9b46ac921b4898 not found: ID does not exist" Oct 30 03:34:38.137102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.137077 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn"] Oct 30 03:34:38.147505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.147486 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-pwltn"] Oct 30 03:34:38.175457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.175404 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6"] Oct 30 03:34:38.179205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.179185 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d80b1-openshift-default-75f5688b99-gtzl6"] Oct 30 03:34:38.187025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:38.187008 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:39.583846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:39.583811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:40.106457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:40.106422 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" path="/var/lib/kubelet/pods/02f9e7b0-9769-4a43-b2b5-8ab40addd8c0/volumes" Oct 30 03:34:40.106991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:40.106968 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" path="/var/lib/kubelet/pods/10abd507-0528-408f-adc7-cdc1fa787d54/volumes" Oct 30 03:34:41.013499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:41.013462 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:41.645628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:41.645598 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:41.745547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:41.745519 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:41.894451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:41.894419 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:41.992250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:41.992150 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:42.753292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:42.753263 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:43.279800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.279774 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:43.393026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.392990 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:43.397522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397490 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr"] Oct 30 03:34:43.397916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397901 4442 cpu_manager.go:414] "containerMap: removing container" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" Oct 30 03:34:43.397916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397916 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397925 4442 cpu_manager.go:414] "containerMap: removing container" podUID="46b8d439-4925-4ca6-87a2-f5514e9f834b" containerName="payload-pre-processing" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397931 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b8d439-4925-4ca6-87a2-f5514e9f834b" containerName="payload-pre-processing" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397943 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2a1196db-cabd-47e3-a02d-4f2a64fc37a6" containerName="maas-api" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397948 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1196db-cabd-47e3-a02d-4f2a64fc37a6" containerName="maas-api" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397956 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c66580b9-aa52-4b09-8be4-77e00592db41" containerName="maas-api" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397961 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c66580b9-aa52-4b09-8be4-77e00592db41" containerName="maas-api" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397974 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dfa35cf8-883a-494f-b949-41cd36f73b41" containerName="payload-processing" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397980 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa35cf8-883a-494f-b949-41cd36f73b41" containerName="payload-processing" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397987 4442 cpu_manager.go:414] "containerMap: removing container" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397991 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.397999 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398003 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398012 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398017 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398026 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fbbdb317-95af-4141-8f1b-55d7eff61a27" containerName="payload-pre-processing" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398030 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbdb317-95af-4141-8f1b-55d7eff61a27" containerName="payload-pre-processing" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398038 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398042 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398049 4442 cpu_manager.go:414] "containerMap: removing container" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" Oct 30 03:34:43.398045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398053 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398062 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a384c723-3eb5-431c-9955-255cf34fdb05" containerName="payload-pre-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398067 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a384c723-3eb5-431c-9955-255cf34fdb05" containerName="payload-pre-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398077 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398081 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398089 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1422630a-fd28-4209-9316-586b74125bce" containerName="payload-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398093 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1422630a-fd28-4209-9316-586b74125bce" containerName="payload-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398151 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbdb317-95af-4141-8f1b-55d7eff61a27" containerName="payload-pre-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398159 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea287de4-a030-4e5e-b106-451729ab7806" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398164 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="10abd507-0528-408f-adc7-cdc1fa787d54" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398171 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a384c723-3eb5-431c-9955-255cf34fdb05" containerName="payload-pre-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398176 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee8ad860-7156-451c-9b66-001c8bbd7107" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398181 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1422630a-fd28-4209-9316-586b74125bce" containerName="payload-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398187 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f928cd-d051-43f2-a6d2-6d40bdc0c635" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398191 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b8d439-4925-4ca6-87a2-f5514e9f834b" containerName="payload-pre-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398196 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1196db-cabd-47e3-a02d-4f2a64fc37a6" containerName="maas-api" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398200 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c66580b9-aa52-4b09-8be4-77e00592db41" containerName="maas-api" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398207 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398212 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa35cf8-883a-494f-b949-41cd36f73b41" containerName="payload-processing" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398218 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="02f9e7b0-9769-4a43-b2b5-8ab40addd8c0" containerName="istio-proxy" Oct 30 03:34:43.398632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.398223 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8235830-dbab-4486-8f1e-3689feb8544e" containerName="istio-proxy" Oct 30 03:34:43.402799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.402783 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.420312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.420290 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr"] Oct 30 03:34:43.428430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.428409 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-da4553\"" Oct 30 03:34:43.428537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.428461 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-da4553-dockercfg-dmljm\"" Oct 30 03:34:43.438694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.438673 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9"] Oct 30 03:34:43.442331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.442312 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.459735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.459709 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9"] Oct 30 03:34:43.525732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.525674 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgfpl\" (UniqueName: \"kubernetes.io/projected/fd668d56-aefe-46e4-aa96-6d02036bac6f-kube-api-access-xgfpl\") pod \"payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.525926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.525822 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt97t\" (UniqueName: \"kubernetes.io/projected/32bdd1c4-68d8-4426-8c82-fc5420f7b535-kube-api-access-vt97t\") pod \"payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.525926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.525875 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bdd1c4-68d8-4426-8c82-fc5420f7b535-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.526008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.525961 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd668d56-aefe-46e4-aa96-6d02036bac6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.626779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.626666 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd668d56-aefe-46e4-aa96-6d02036bac6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.626779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.626731 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgfpl\" (UniqueName: \"kubernetes.io/projected/fd668d56-aefe-46e4-aa96-6d02036bac6f-kube-api-access-xgfpl\") pod \"payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.627006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.626879 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt97t\" (UniqueName: \"kubernetes.io/projected/32bdd1c4-68d8-4426-8c82-fc5420f7b535-kube-api-access-vt97t\") pod \"payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.627006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.626922 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bdd1c4-68d8-4426-8c82-fc5420f7b535-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.627429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.627406 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd668d56-aefe-46e4-aa96-6d02036bac6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.627615 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.627596 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bdd1c4-68d8-4426-8c82-fc5420f7b535-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.645747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.645725 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt97t\" (UniqueName: \"kubernetes.io/projected/32bdd1c4-68d8-4426-8c82-fc5420f7b535-kube-api-access-vt97t\") pod \"payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.645880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.645768 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgfpl\" (UniqueName: \"kubernetes.io/projected/fd668d56-aefe-46e4-aa96-6d02036bac6f-kube-api-access-xgfpl\") pod \"payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.716125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.716101 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:43.756849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.756825 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:43.797634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.797599 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk"] Oct 30 03:34:43.798065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:43.797833 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" containerID="cri-o://6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4" gracePeriod=30 Oct 30 03:34:44.016426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.016395 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr"] Oct 30 03:34:44.017443 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:34:44.017413 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32bdd1c4_68d8_4426_8c82_fc5420f7b535.slice/crio-c6cb4c67552383ef49ef5effb3de454d3c75487d9c710cb6baaa15cc353fc38c WatchSource:0}: Error finding container c6cb4c67552383ef49ef5effb3de454d3c75487d9c710cb6baaa15cc353fc38c: Status 404 returned error can't find the container with id c6cb4c67552383ef49ef5effb3de454d3c75487d9c710cb6baaa15cc353fc38c Oct 30 03:34:44.076361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.076339 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9"] Oct 30 03:34:44.076989 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:34:44.076967 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd668d56_aefe_46e4_aa96_6d02036bac6f.slice/crio-749a8937c550b220bfdf938edfffadd5b9b0f45b8f994d8e975c97bb7674428e WatchSource:0}: Error finding container 749a8937c550b220bfdf938edfffadd5b9b0f45b8f994d8e975c97bb7674428e: Status 404 returned error can't find the container with id 749a8937c550b220bfdf938edfffadd5b9b0f45b8f994d8e975c97bb7674428e Oct 30 03:34:44.130988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.130962 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" event={"ID":"fd668d56-aefe-46e4-aa96-6d02036bac6f","Type":"ContainerStarted","Data":"749a8937c550b220bfdf938edfffadd5b9b0f45b8f994d8e975c97bb7674428e"} Oct 30 03:34:44.132593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.132568 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" event={"ID":"32bdd1c4-68d8-4426-8c82-fc5420f7b535","Type":"ContainerStarted","Data":"a3c17e29e03104c1974eb7620f5751118fa3925131eec74a00f8652ad8c309f5"} Oct 30 03:34:44.132713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.132617 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" event={"ID":"32bdd1c4-68d8-4426-8c82-fc5420f7b535","Type":"ContainerStarted","Data":"c6cb4c67552383ef49ef5effb3de454d3c75487d9c710cb6baaa15cc353fc38c"} Oct 30 03:34:44.132713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.132666 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:44.168266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.168201 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" podStartSLOduration=1.168184807 podStartE2EDuration="1.168184807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:44.16683471 +0000 UTC m=+1908.169187933" watchObservedRunningTime="2026-10-30 03:34:44.168184807 +0000 UTC m=+1908.170538007" Oct 30 03:34:44.310768 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.310646 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:44.737771 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.737651 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:44.830612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.830577 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:44.851073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.851046 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn"] Oct 30 03:34:44.854707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.854692 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:44.873064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.873026 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-1a1b45\"" Oct 30 03:34:44.873203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.873090 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-1a1b45-dockercfg-qhb6b\"" Oct 30 03:34:44.873203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.873117 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn"] Oct 30 03:34:44.939764 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.939699 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gk8t\" (UniqueName: \"kubernetes.io/projected/7e4351d9-3c06-4da5-96a8-11ffd833984a-kube-api-access-7gk8t\") pod \"payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:44.940003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:44.939937 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4351d9-3c06-4da5-96a8-11ffd833984a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:45.040739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.040702 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gk8t\" (UniqueName: \"kubernetes.io/projected/7e4351d9-3c06-4da5-96a8-11ffd833984a-kube-api-access-7gk8t\") pod \"payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:45.040938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.040812 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4351d9-3c06-4da5-96a8-11ffd833984a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:45.041555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.041530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4351d9-3c06-4da5-96a8-11ffd833984a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:45.058188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.058159 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gk8t\" (UniqueName: \"kubernetes.io/projected/7e4351d9-3c06-4da5-96a8-11ffd833984a-kube-api-access-7gk8t\") pod \"payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:45.138879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.138846 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" event={"ID":"fd668d56-aefe-46e4-aa96-6d02036bac6f","Type":"ContainerStarted","Data":"d140c6dbf32eede2b07b23748914d78402a5cd4031abc844ac49c0048c95a0d0"} Oct 30 03:34:45.169034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.169008 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:45.176784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.176714 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" podStartSLOduration=2.176697067 podStartE2EDuration="2.176697067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:45.173816245 +0000 UTC m=+1909.176169445" watchObservedRunningTime="2026-10-30 03:34:45.176697067 +0000 UTC m=+1909.179050264" Oct 30 03:34:45.469522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.469496 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn"] Oct 30 03:34:45.471436 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:34:45.471401 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4351d9_3c06_4da5_96a8_11ffd833984a.slice/crio-39b917a88e32825d6a10dfc1f77be13dfbfc7fbf03875b9dc795c1c9508f6fa0 WatchSource:0}: Error finding container 39b917a88e32825d6a10dfc1f77be13dfbfc7fbf03875b9dc795c1c9508f6fa0: Status 404 returned error can't find the container with id 39b917a88e32825d6a10dfc1f77be13dfbfc7fbf03875b9dc795c1c9508f6fa0 Oct 30 03:34:45.880966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:45.880937 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:46.146165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:46.146081 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" event={"ID":"7e4351d9-3c06-4da5-96a8-11ffd833984a","Type":"ContainerStarted","Data":"bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00"} Oct 30 03:34:46.146165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:46.146127 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" event={"ID":"7e4351d9-3c06-4da5-96a8-11ffd833984a","Type":"ContainerStarted","Data":"39b917a88e32825d6a10dfc1f77be13dfbfc7fbf03875b9dc795c1c9508f6fa0"} Oct 30 03:34:46.146165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:46.146156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:46.146392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:46.146346 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:46.184924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:46.184881 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" podStartSLOduration=2.184867747 podStartE2EDuration="2.184867747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:46.184165515 +0000 UTC m=+1910.186518732" watchObservedRunningTime="2026-10-30 03:34:46.184867747 +0000 UTC m=+1910.187220943" Oct 30 03:34:47.057952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:47.057924 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:47.394732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:47.394655 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:48.078356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:48.078326 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:48.786063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:48.786014 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:49.069670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.069645 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:34:49.085364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085328 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-data\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085380 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-certs\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085405 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-credential-socket\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085443 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-envoy\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085470 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-socket\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085510 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istiod-ca-cert\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085559 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-kube-api-access-v7kkr\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-kube-api-access-v7kkr\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085617 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-podinfo\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085612 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-data" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085717 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-credential-socket" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:49.085734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085631 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-certs" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:49.086255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.085835 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-socket" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:49.086255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.086030 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.086255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.086052 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.086255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.086068 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.086255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.086082 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.086255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.086091 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istiod-ca-cert" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:34:49.087819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.087798 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-kube-api-access-v7kkr" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "kube-api-access-v7kkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:49.088371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.088349 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-envoy" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:34:49.088455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.088352 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-podinfo" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:34:49.163878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.163845 4442 generic.go:356] "Generic (PLEG): container finished" podID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerID="6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4" exitCode=0 Oct 30 03:34:49.164046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.163900 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" event={"ID":"0baf7f8a-0670-475c-9f9b-be6addc4cb5f","Type":"ContainerDied","Data":"6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4"} Oct 30 03:34:49.164046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.163966 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" event={"ID":"0baf7f8a-0670-475c-9f9b-be6addc4cb5f","Type":"ContainerDied","Data":"8af51baf553a0272ad6e3d9611a38861b3e6c65075695da2aaf4be7a36cefc89"} Oct 30 03:34:49.164046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.163909 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk" Oct 30 03:34:49.164046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.163997 4442 scope.go:122] "RemoveContainer" containerID="6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4" Oct 30 03:34:49.173354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.173327 4442 scope.go:122] "RemoveContainer" containerID="6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4" Oct 30 03:34:49.173631 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:34:49.173613 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4\": container with ID starting with 6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4 not found: ID does not exist" containerID="6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4" Oct 30 03:34:49.173698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.173641 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4"} err="failed to get container status \"6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4\": rpc error: code = NotFound desc = could not find container \"6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4\": container with ID starting with 6e8c95a1910f58d93e1d3c288b507b89d2df583b889d57f778e05ae6299f3dd4 not found: ID does not exist" Oct 30 03:34:49.186715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.186695 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-token\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-token\") pod \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\" (UID: \"0baf7f8a-0670-475c-9f9b-be6addc4cb5f\") " Oct 30 03:34:49.186993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.186976 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.186993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.186998 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.187137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.187019 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7kkr\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-kube-api-access-v7kkr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.187137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.187039 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.188688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.188665 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-token" pod "0baf7f8a-0670-475c-9f9b-be6addc4cb5f" (UID: "0baf7f8a-0670-475c-9f9b-be6addc4cb5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:34:49.287530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.287492 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0baf7f8a-0670-475c-9f9b-be6addc4cb5f-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:34:49.506871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.506839 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk"] Oct 30 03:34:49.517442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:49.517415 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-wgssk"] Oct 30 03:34:50.046040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.046013 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:50.106946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.106874 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" path="/var/lib/kubelet/pods/0baf7f8a-0670-475c-9f9b-be6addc4cb5f/volumes" Oct 30 03:34:50.183049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.183013 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr"] Oct 30 03:34:50.183469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.183453 4442 cpu_manager.go:414] "containerMap: removing container" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" Oct 30 03:34:50.183469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.183468 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" Oct 30 03:34:50.183611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.183559 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baf7f8a-0670-475c-9f9b-be6addc4cb5f" containerName="istio-proxy" Oct 30 03:34:50.186404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.186389 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.196049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.196023 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.196150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.196084 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15707df2-3b1c-43a8-8a99-b1f10231cb00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.196227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.196172 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.196287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.196230 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfbd\" (UniqueName: \"kubernetes.io/projected/15707df2-3b1c-43a8-8a99-b1f10231cb00-kube-api-access-5wfbd\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.213673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.213648 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-da4553-serving-cert\"" Oct 30 03:34:50.217390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.217357 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr"] Oct 30 03:34:50.252668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.252643 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:50.297119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.297090 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15707df2-3b1c-43a8-8a99-b1f10231cb00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.297271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.297126 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.297271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.297158 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfbd\" (UniqueName: \"kubernetes.io/projected/15707df2-3b1c-43a8-8a99-b1f10231cb00-kube-api-access-5wfbd\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.297383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.297274 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.297794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.297745 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15707df2-3b1c-43a8-8a99-b1f10231cb00-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.299790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.299738 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.299885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.299791 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.315007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.314984 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfbd\" (UniqueName: \"kubernetes.io/projected/15707df2-3b1c-43a8-8a99-b1f10231cb00-kube-api-access-5wfbd\") pod \"maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.330007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.329986 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:50.430922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.430828 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:50.499453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.499421 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:50.800160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.800131 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr"] Oct 30 03:34:50.801600 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:34:50.801571 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15707df2_3b1c_43a8_8a99_b1f10231cb00.slice/crio-5bce4891c47b0b826d7cb034e2afbd69fba45588ec94d6e5dffcae4c5316984e WatchSource:0}: Error finding container 5bce4891c47b0b826d7cb034e2afbd69fba45588ec94d6e5dffcae4c5316984e: Status 404 returned error can't find the container with id 5bce4891c47b0b826d7cb034e2afbd69fba45588ec94d6e5dffcae4c5316984e Oct 30 03:34:50.802968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:50.802951 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:34:51.176198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.176118 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" event={"ID":"15707df2-3b1c-43a8-8a99-b1f10231cb00","Type":"ContainerStarted","Data":"5bce4891c47b0b826d7cb034e2afbd69fba45588ec94d6e5dffcae4c5316984e"} Oct 30 03:34:51.631809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.631772 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d"] Oct 30 03:34:51.635286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.635271 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.661789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.661487 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-1a1b45-serving-cert\"" Oct 30 03:34:51.663475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.663450 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d"] Oct 30 03:34:51.703759 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.703722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:51.717279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.717254 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd79d028-431b-43e4-bc99-e42209568e78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.717425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.717288 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.717425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.717395 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9h2\" (UniqueName: \"kubernetes.io/projected/dd79d028-431b-43e4-bc99-e42209568e78-kube-api-access-zm9h2\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.717503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.717424 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.803447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.803424 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:51.828721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.828678 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9h2\" (UniqueName: \"kubernetes.io/projected/dd79d028-431b-43e4-bc99-e42209568e78-kube-api-access-zm9h2\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.839827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.838337 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.839827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.838506 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd79d028-431b-43e4-bc99-e42209568e78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.839827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.838538 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.839827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.839548 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd79d028-431b-43e4-bc99-e42209568e78-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.841936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.841897 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.842402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.842377 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.847825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.847798 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9h2\" (UniqueName: \"kubernetes.io/projected/dd79d028-431b-43e4-bc99-e42209568e78-kube-api-access-zm9h2\") pod \"maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:51.948703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:51.948629 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:52.182550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:52.182512 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" event={"ID":"15707df2-3b1c-43a8-8a99-b1f10231cb00","Type":"ContainerStarted","Data":"d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5"} Oct 30 03:34:52.182974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:52.182636 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:52.220119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:52.220036 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" podStartSLOduration=1.769473158 podStartE2EDuration="2.220022276s" totalImagesPullingTime="450.549118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:52.217975751 +0000 UTC m=+1916.220328960" watchObservedRunningTime="2026-10-30 03:34:52.220022276 +0000 UTC m=+1916.222375473" Oct 30 03:34:52.267276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:52.267249 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d"] Oct 30 03:34:52.268871 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:34:52.268841 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd79d028_431b_43e4_bc99_e42209568e78.slice/crio-49cbc14c1505414601504e5933d13c3a6cf04acfb8768249bc63e1271669d91f WatchSource:0}: Error finding container 49cbc14c1505414601504e5933d13c3a6cf04acfb8768249bc63e1271669d91f: Status 404 returned error can't find the container with id 49cbc14c1505414601504e5933d13c3a6cf04acfb8768249bc63e1271669d91f Oct 30 03:34:53.081293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.081262 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:53.189992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.189956 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" event={"ID":"dd79d028-431b-43e4-bc99-e42209568e78","Type":"ContainerStarted","Data":"ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418"} Oct 30 03:34:53.190419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.190015 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" event={"ID":"dd79d028-431b-43e4-bc99-e42209568e78","Type":"ContainerStarted","Data":"49cbc14c1505414601504e5933d13c3a6cf04acfb8768249bc63e1271669d91f"} Oct 30 03:34:53.190419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.190190 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:53.231384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.231334 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" podStartSLOduration=1.8832688499999999 podStartE2EDuration="2.231321213s" totalImagesPullingTime="348.052363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:53.228872352 +0000 UTC m=+1917.231225550" watchObservedRunningTime="2026-10-30 03:34:53.231321213 +0000 UTC m=+1917.233674440" Oct 30 03:34:53.277349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.277320 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:53.607496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.607458 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:53.848467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:53.848430 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:54.802058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:54.802017 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:54.880267 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:34:54.880229 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:34:55.141715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:55.141637 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 03:34:56.271239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:56.271210 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:57.155396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:57.155304 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 03:34:57.155541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:57.155437 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 03:34:57.741069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:57.741033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:57.843107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:57.843070 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:58.195872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:58.195845 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 03:34:59.202344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:59.202311 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 03:34:59.399621 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:59.399576 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:34:59.520715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:34:59.520670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:00.098025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:00.097990 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:35:01.025907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:01.025871 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:01.409720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:01.409621 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:01.912538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:01.912504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:02.076314 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:02.076277 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:02.577947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:02.577915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:04.015567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:04.015537 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:05.616045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:05.616015 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:05.805507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:05.805469 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:05.898705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:05.898626 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:06.940253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:06.940219 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:07.043005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.042977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:07.398113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.398076 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:07.497389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.497346 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:07.576077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.576043 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf"] Oct 30 03:35:07.581137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.581121 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.605135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.605117 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6ad168\"" Oct 30 03:35:07.605274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.605191 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6ad168-dockercfg-84hss\"" Oct 30 03:35:07.607705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.607686 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf"] Oct 30 03:35:07.622277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.622255 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t"] Oct 30 03:35:07.624983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.624958 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:07.626243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.626227 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:07.651161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.651109 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t"] Oct 30 03:35:07.714380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.714036 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.714380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.714093 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891f78f5-3792-4618-904b-2e1a6ae804cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:07.714380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.714150 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncr8m\" (UniqueName: \"kubernetes.io/projected/891f78f5-3792-4618-904b-2e1a6ae804cc-kube-api-access-ncr8m\") pod \"payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:07.714380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.714195 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5zp\" (UniqueName: \"kubernetes.io/projected/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-kube-api-access-pr5zp\") pod \"payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.815513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.815478 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.815513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.815522 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891f78f5-3792-4618-904b-2e1a6ae804cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:07.815740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.815555 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncr8m\" (UniqueName: \"kubernetes.io/projected/891f78f5-3792-4618-904b-2e1a6ae804cc-kube-api-access-ncr8m\") pod \"payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:07.815740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.815579 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5zp\" (UniqueName: \"kubernetes.io/projected/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-kube-api-access-pr5zp\") pod \"payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.816209 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.816189 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.816305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.816288 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891f78f5-3792-4618-904b-2e1a6ae804cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:07.837654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.837619 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq"] Oct 30 03:35:07.842365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.842340 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.849100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.849067 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncr8m\" (UniqueName: \"kubernetes.io/projected/891f78f5-3792-4618-904b-2e1a6ae804cc-kube-api-access-ncr8m\") pod \"payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:07.849833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.849809 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5zp\" (UniqueName: \"kubernetes.io/projected/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-kube-api-access-pr5zp\") pod \"payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.862783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.860312 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e1a0c2-openshift-default-dockercfg-njtkh\"" Oct 30 03:35:07.863606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.863582 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq"] Oct 30 03:35:07.887092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.887060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:07.894024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.893994 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:07.916220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916194 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916305 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpqz\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-kube-api-access-cqpqz\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916363 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916385 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916491 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916645 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916621 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb022099-cfd2-41b8-aabc-c6cb77f40112-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916721 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.916849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.916791 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:07.942403 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:07.942376 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:08.005811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.005223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:08.018249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018213 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018273 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018367 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb022099-cfd2-41b8-aabc-c6cb77f40112-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018434 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018461 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018499 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018558 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpqz\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-kube-api-access-cqpqz\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018608 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.018849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.018632 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.020556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.019083 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-certs\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.020556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.019620 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-data\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.020556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.019912 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-credential-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.020556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.020420 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-socket\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.020556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.020510 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb022099-cfd2-41b8-aabc-c6cb77f40112-istiod-ca-cert\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.024006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.023963 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-envoy\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.026401 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.026365 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-podinfo\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.045593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.045570 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpqz\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-kube-api-access-cqpqz\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.047482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.047449 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-token\") pod \"e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.132697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.132649 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:08.172829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.172740 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:08.203529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.203500 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf"] Oct 30 03:35:08.204697 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:08.204668 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7523a6_6cad_44c1_9ae8_0005ea2ae558.slice/crio-8c3378c8dad59547e86f577b846fe9de2ea567e2c2faf03f0669b37f439e772e WatchSource:0}: Error finding container 8c3378c8dad59547e86f577b846fe9de2ea567e2c2faf03f0669b37f439e772e: Status 404 returned error can't find the container with id 8c3378c8dad59547e86f577b846fe9de2ea567e2c2faf03f0669b37f439e772e Oct 30 03:35:08.260851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.260819 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:08.276834 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:08.276802 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891f78f5_3792_4618_904b_2e1a6ae804cc.slice/crio-0bffeeaaf6ad92094b7b86ceb9eb7a81413c967d526ae229df362a2c7b7f12a8 WatchSource:0}: Error finding container 0bffeeaaf6ad92094b7b86ceb9eb7a81413c967d526ae229df362a2c7b7f12a8: Status 404 returned error can't find the container with id 0bffeeaaf6ad92094b7b86ceb9eb7a81413c967d526ae229df362a2c7b7f12a8 Oct 30 03:35:08.278919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.278883 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t"] Oct 30 03:35:08.279043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.278998 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" event={"ID":"ac7523a6-6cad-44c1-9ae8-0005ea2ae558","Type":"ContainerStarted","Data":"8c3378c8dad59547e86f577b846fe9de2ea567e2c2faf03f0669b37f439e772e"} Oct 30 03:35:08.479951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.479926 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq"] Oct 30 03:35:08.481305 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:08.481271 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb022099_cfd2_41b8_aabc_c6cb77f40112.slice/crio-39c6894f56a0124493642b972a6f6db59375839d53e794cb4735ac68ebefdc45 WatchSource:0}: Error finding container 39c6894f56a0124493642b972a6f6db59375839d53e794cb4735ac68ebefdc45: Status 404 returned error can't find the container with id 39c6894f56a0124493642b972a6f6db59375839d53e794cb4735ac68ebefdc45 Oct 30 03:35:08.483962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.483928 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:35:08.484061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.483994 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:35:08.484061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:08.484024 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:35:09.093974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.093945 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:09.287159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.287069 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" event={"ID":"891f78f5-3792-4618-904b-2e1a6ae804cc","Type":"ContainerStarted","Data":"de2902ba476da8505e934c4411ee62bda97fdf792d4629b24d0f4ffe169d9bbd"} Oct 30 03:35:09.287159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.287141 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:09.287364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.287167 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" event={"ID":"891f78f5-3792-4618-904b-2e1a6ae804cc","Type":"ContainerStarted","Data":"0bffeeaaf6ad92094b7b86ceb9eb7a81413c967d526ae229df362a2c7b7f12a8"} Oct 30 03:35:09.288869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.288843 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" event={"ID":"cb022099-cfd2-41b8-aabc-c6cb77f40112","Type":"ContainerStarted","Data":"f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff"} Oct 30 03:35:09.289006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.288896 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" event={"ID":"cb022099-cfd2-41b8-aabc-c6cb77f40112","Type":"ContainerStarted","Data":"39c6894f56a0124493642b972a6f6db59375839d53e794cb4735ac68ebefdc45"} Oct 30 03:35:09.290351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.290325 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" event={"ID":"ac7523a6-6cad-44c1-9ae8-0005ea2ae558","Type":"ContainerStarted","Data":"513ab0611dfcf1a02d96ed4ebe7194c5ec2a67aed8e91bc55da6769bad991a88"} Oct 30 03:35:09.290466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.290448 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:09.324227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.324181 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" podStartSLOduration=2.32416876 podStartE2EDuration="2.32416876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:09.322121085 +0000 UTC m=+1933.324474281" watchObservedRunningTime="2026-10-30 03:35:09.32416876 +0000 UTC m=+1933.326521957" Oct 30 03:35:09.363747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.363613 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podStartSLOduration=2.363602283 podStartE2EDuration="2.363602283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:09.362253682 +0000 UTC m=+1933.364606889" watchObservedRunningTime="2026-10-30 03:35:09.363602283 +0000 UTC m=+1933.365955479" Oct 30 03:35:09.398285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:09.398238 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" podStartSLOduration=2.398225599 podStartE2EDuration="2.398225599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:09.396996313 +0000 UTC m=+1933.399349509" watchObservedRunningTime="2026-10-30 03:35:09.398225599 +0000 UTC m=+1933.400578796" Oct 30 03:35:10.173615 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:10.173582 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:10.178928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:10.178901 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:10.296088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:10.296055 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:10.296902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:10.296878 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:10.720885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:10.720852 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:10.808454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:10.808421 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:11.819485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:11.819450 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:11.925716 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:11.925684 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:12.054073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:12.054031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:12.216377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:12.216298 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:12.614350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:12.614312 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:13.452144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.452105 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:13.548523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.548494 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:13.577340 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.577311 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm"] Oct 30 03:35:13.581379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.581358 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:13.596039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.596019 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm"] Oct 30 03:35:13.604996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.604975 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e1a0c2\"" Oct 30 03:35:13.605095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.604981 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e1a0c2-dockercfg-hpqrp\"" Oct 30 03:35:13.723828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.705957 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn4jz\" (UniqueName: \"kubernetes.io/projected/5d629a0a-edeb-4351-9b6d-692d09939b9f-kube-api-access-mn4jz\") pod \"payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:13.723828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.706017 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d629a0a-edeb-4351-9b6d-692d09939b9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:13.807569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.807536 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn4jz\" (UniqueName: \"kubernetes.io/projected/5d629a0a-edeb-4351-9b6d-692d09939b9f-kube-api-access-mn4jz\") pod \"payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:13.807767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.807574 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d629a0a-edeb-4351-9b6d-692d09939b9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:13.808302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.808279 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d629a0a-edeb-4351-9b6d-692d09939b9f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:13.828540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.828511 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn4jz\" (UniqueName: \"kubernetes.io/projected/5d629a0a-edeb-4351-9b6d-692d09939b9f-kube-api-access-mn4jz\") pod \"payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:13.895369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:13.895333 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:14.194513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:14.194486 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm"] Oct 30 03:35:14.195992 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:14.195964 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d629a0a_edeb_4351_9b6d_692d09939b9f.slice/crio-1b114e0c491faa7bb8445b1811ba4db4ac375162393a2caf9f4942c498ed19dc WatchSource:0}: Error finding container 1b114e0c491faa7bb8445b1811ba4db4ac375162393a2caf9f4942c498ed19dc: Status 404 returned error can't find the container with id 1b114e0c491faa7bb8445b1811ba4db4ac375162393a2caf9f4942c498ed19dc Oct 30 03:35:14.324292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:14.324263 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" event={"ID":"5d629a0a-edeb-4351-9b6d-692d09939b9f","Type":"ContainerStarted","Data":"35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603"} Oct 30 03:35:14.324467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:14.324310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" event={"ID":"5d629a0a-edeb-4351-9b6d-692d09939b9f","Type":"ContainerStarted","Data":"1b114e0c491faa7bb8445b1811ba4db4ac375162393a2caf9f4942c498ed19dc"} Oct 30 03:35:14.324467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:14.324344 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:14.360968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:14.360919 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" podStartSLOduration=1.360904298 podStartE2EDuration="1.360904298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:14.359733142 +0000 UTC m=+1938.362086341" watchObservedRunningTime="2026-10-30 03:35:14.360904298 +0000 UTC m=+1938.363257495" Oct 30 03:35:14.407935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:14.407906 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:14.966936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:14.966895 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:15.079764 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.079715 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv"] Oct 30 03:35:15.083600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.083583 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.086598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.086570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:15.109636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.108988 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-547a78\"" Oct 30 03:35:15.109636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.109168 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-547a78-dockercfg-8r22p\"" Oct 30 03:35:15.112881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.112861 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv"] Oct 30 03:35:15.233159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.233070 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvfs\" (UniqueName: \"kubernetes.io/projected/2b37ede5-6f78-44aa-9211-4e23d05941c2-kube-api-access-psvfs\") pod \"payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.233436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.233412 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b37ede5-6f78-44aa-9211-4e23d05941c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.334541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.334509 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvfs\" (UniqueName: \"kubernetes.io/projected/2b37ede5-6f78-44aa-9211-4e23d05941c2-kube-api-access-psvfs\") pod \"payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.334710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.334545 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b37ede5-6f78-44aa-9211-4e23d05941c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.335232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.335214 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b37ede5-6f78-44aa-9211-4e23d05941c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.352365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.352330 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvfs\" (UniqueName: \"kubernetes.io/projected/2b37ede5-6f78-44aa-9211-4e23d05941c2-kube-api-access-psvfs\") pod \"payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.397052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.397025 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:15.693933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.693907 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv"] Oct 30 03:35:15.695721 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:15.695692 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b37ede5_6f78_44aa_9211_4e23d05941c2.slice/crio-9ca5b7042ff9934a9c179549926b7581128cb179a55f94902a63450c14c2d5ca WatchSource:0}: Error finding container 9ca5b7042ff9934a9c179549926b7581128cb179a55f94902a63450c14c2d5ca: Status 404 returned error can't find the container with id 9ca5b7042ff9934a9c179549926b7581128cb179a55f94902a63450c14c2d5ca Oct 30 03:35:15.978353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:15.978279 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:16.336831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:16.336792 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" event={"ID":"2b37ede5-6f78-44aa-9211-4e23d05941c2","Type":"ContainerStarted","Data":"993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7"} Oct 30 03:35:16.337016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:16.336855 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" event={"ID":"2b37ede5-6f78-44aa-9211-4e23d05941c2","Type":"ContainerStarted","Data":"9ca5b7042ff9934a9c179549926b7581128cb179a55f94902a63450c14c2d5ca"} Oct 30 03:35:16.337016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:16.336932 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:16.372466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:16.372418 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" podStartSLOduration=1.372405142 podStartE2EDuration="1.372405142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:16.371636985 +0000 UTC m=+1940.373990184" watchObservedRunningTime="2026-10-30 03:35:16.372405142 +0000 UTC m=+1940.374758339" Oct 30 03:35:17.509006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:17.508975 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:18.969494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:18.969467 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:20.298733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:20.298701 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 03:35:20.299246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:20.298798 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 03:35:20.440917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:20.440889 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:20.526999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:20.526965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:20.714862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:20.714776 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:35:20.719452 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:20.719429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:22.074649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:22.074620 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:23.573066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:23.573033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:24.860288 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:35:24.860248 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:35:25.034420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:25.034360 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:25.332515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:25.332490 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 03:35:26.544034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.544002 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96"] Oct 30 03:35:26.548013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.547997 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.573673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.573643 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e1a0c2-serving-cert\"" Oct 30 03:35:26.578266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.578236 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96"] Oct 30 03:35:26.624066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.624036 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:26.668617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.668582 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtw8\" (UniqueName: \"kubernetes.io/projected/ac4a8fc7-1451-454a-979d-e1f05a3e1570-kube-api-access-phtw8\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.668775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.668641 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.668862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.668839 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4a8fc7-1451-454a-979d-e1f05a3e1570-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.669008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.668985 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.724043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.724013 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:26.780936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.780901 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4a8fc7-1451-454a-979d-e1f05a3e1570-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.781312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.781285 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.785971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.785942 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtw8\" (UniqueName: \"kubernetes.io/projected/ac4a8fc7-1451-454a-979d-e1f05a3e1570-kube-api-access-phtw8\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.786555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.781489 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4a8fc7-1451-454a-979d-e1f05a3e1570-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.786712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.786695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.791361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.791337 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.795677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.795591 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.804708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.804682 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtw8\" (UniqueName: \"kubernetes.io/projected/ac4a8fc7-1451-454a-979d-e1f05a3e1570-kube-api-access-phtw8\") pod \"maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:26.824113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.824088 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:26.862114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:26.862095 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:27.159659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:27.159631 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96"] Oct 30 03:35:27.160823 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:27.160799 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4a8fc7_1451_454a_979d_e1f05a3e1570.slice/crio-5a2979a2f746f0ab8f7ac6a6b3ecd1b0377630c70fc65abe609fff33292b1c44 WatchSource:0}: Error finding container 5a2979a2f746f0ab8f7ac6a6b3ecd1b0377630c70fc65abe609fff33292b1c44: Status 404 returned error can't find the container with id 5a2979a2f746f0ab8f7ac6a6b3ecd1b0377630c70fc65abe609fff33292b1c44 Oct 30 03:35:27.345883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:27.345850 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 03:35:27.403876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:27.403827 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" event={"ID":"ac4a8fc7-1451-454a-979d-e1f05a3e1570","Type":"ContainerStarted","Data":"5a2979a2f746f0ab8f7ac6a6b3ecd1b0377630c70fc65abe609fff33292b1c44"} Oct 30 03:35:28.168609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:28.168575 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:28.268595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:28.268564 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:28.369167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:28.369130 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:28.410649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:28.410606 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" event={"ID":"ac4a8fc7-1451-454a-979d-e1f05a3e1570","Type":"ContainerStarted","Data":"a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a"} Oct 30 03:35:28.410826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:28.410711 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:28.450977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:28.450888 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" podStartSLOduration=2.109532801 podStartE2EDuration="2.450875142s" totalImagesPullingTime="341.342341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:28.449050744 +0000 UTC m=+1952.451403941" watchObservedRunningTime="2026-10-30 03:35:28.450875142 +0000 UTC m=+1952.453228338" Oct 30 03:35:29.688598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:29.688565 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:31.113240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:31.113194 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:32.696512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:32.696479 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:33.326811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:33.326775 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:33.843540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:33.843505 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:34.205535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:34.205462 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:34.309950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:34.309918 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:34.428953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:34.428906 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 03:35:34.558573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:34.558544 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:35.689116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:35.689072 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:37.171051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:37.171014 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:38.253280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:38.253248 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:38.355704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:38.355676 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:38.713778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:38.713654 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:39.059425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:39.059390 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:35:40.049700 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.049667 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:40.115813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.115780 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p"] Oct 30 03:35:40.122018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.121988 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.133920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.133884 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p"] Oct 30 03:35:40.138584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.138555 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-852107\"" Oct 30 03:35:40.148197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.147912 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl9qp\" (UniqueName: \"kubernetes.io/projected/845e7f17-4711-4381-9302-7a3568a6e48d-kube-api-access-xl9qp\") pod \"payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.148197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.147984 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e7f17-4711-4381-9302-7a3568a6e48d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.149560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.149536 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-852107-dockercfg-6cz7w\"" Oct 30 03:35:40.154062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.154028 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6"] Oct 30 03:35:40.157893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.157862 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.158189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.158167 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:40.177878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.177844 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6"] Oct 30 03:35:40.253778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.252087 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl9qp\" (UniqueName: \"kubernetes.io/projected/845e7f17-4711-4381-9302-7a3568a6e48d-kube-api-access-xl9qp\") pod \"payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.253778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.252181 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e7f17-4711-4381-9302-7a3568a6e48d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.254062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.253965 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e7f17-4711-4381-9302-7a3568a6e48d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.294776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.293810 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl9qp\" (UniqueName: \"kubernetes.io/projected/845e7f17-4711-4381-9302-7a3568a6e48d-kube-api-access-xl9qp\") pod \"payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.353659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.353578 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrz6n\" (UniqueName: \"kubernetes.io/projected/193ae444-3be2-447a-a65e-90d467725718-kube-api-access-nrz6n\") pod \"payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.353841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.353702 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ae444-3be2-447a-a65e-90d467725718-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.453267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.453228 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:40.454548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.454521 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ae444-3be2-447a-a65e-90d467725718-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.454659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.454647 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrz6n\" (UniqueName: \"kubernetes.io/projected/193ae444-3be2-447a-a65e-90d467725718-kube-api-access-nrz6n\") pod \"payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.455243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.455225 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ae444-3be2-447a-a65e-90d467725718-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.472445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.472421 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrz6n\" (UniqueName: \"kubernetes.io/projected/193ae444-3be2-447a-a65e-90d467725718-kube-api-access-nrz6n\") pod \"payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.473345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.473292 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:40.475793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.475771 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:40.808385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.808341 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p"] Oct 30 03:35:40.811642 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:40.811611 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845e7f17_4711_4381_9302_7a3568a6e48d.slice/crio-1eb22f41274cb6a414bffb1cdf3fabb20f2fe0affed3734925eceb472f7a2aec WatchSource:0}: Error finding container 1eb22f41274cb6a414bffb1cdf3fabb20f2fe0affed3734925eceb472f7a2aec: Status 404 returned error can't find the container with id 1eb22f41274cb6a414bffb1cdf3fabb20f2fe0affed3734925eceb472f7a2aec Oct 30 03:35:40.858507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:40.858481 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6"] Oct 30 03:35:40.859316 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:40.859292 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193ae444_3be2_447a_a65e_90d467725718.slice/crio-f1c6f5a1d2d2a8dc4488c1f0299d8475e627570eceb1a820092b4d18002e9957 WatchSource:0}: Error finding container f1c6f5a1d2d2a8dc4488c1f0299d8475e627570eceb1a820092b4d18002e9957: Status 404 returned error can't find the container with id f1c6f5a1d2d2a8dc4488c1f0299d8475e627570eceb1a820092b4d18002e9957 Oct 30 03:35:41.327072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.327043 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:41.516660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.516623 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" event={"ID":"193ae444-3be2-447a-a65e-90d467725718","Type":"ContainerStarted","Data":"edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639"} Oct 30 03:35:41.516879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.516679 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" event={"ID":"193ae444-3be2-447a-a65e-90d467725718","Type":"ContainerStarted","Data":"f1c6f5a1d2d2a8dc4488c1f0299d8475e627570eceb1a820092b4d18002e9957"} Oct 30 03:35:41.516879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.516721 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:41.518317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.518294 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" event={"ID":"845e7f17-4711-4381-9302-7a3568a6e48d","Type":"ContainerStarted","Data":"1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471"} Oct 30 03:35:41.518448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.518336 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" event={"ID":"845e7f17-4711-4381-9302-7a3568a6e48d","Type":"ContainerStarted","Data":"1eb22f41274cb6a414bffb1cdf3fabb20f2fe0affed3734925eceb472f7a2aec"} Oct 30 03:35:41.518448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.518368 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:41.554220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.554173 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" podStartSLOduration=1.554160763 podStartE2EDuration="1.554160763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:41.551729618 +0000 UTC m=+1965.554082826" watchObservedRunningTime="2026-10-30 03:35:41.554160763 +0000 UTC m=+1965.556513959" Oct 30 03:35:41.608957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.608846 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" podStartSLOduration=1.608832194 podStartE2EDuration="1.608832194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:41.608150224 +0000 UTC m=+1965.610503423" watchObservedRunningTime="2026-10-30 03:35:41.608832194 +0000 UTC m=+1965.611185390" Oct 30 03:35:41.839909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:41.839873 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:42.208051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:42.208017 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:42.375703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:42.375632 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:42.649219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:42.649185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:43.767970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:43.767941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:45.408237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:45.408210 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:46.548868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:46.548835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:46.646066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:46.646029 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:47.122708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:47.122673 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:47.238435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:47.238401 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:48.209339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.209305 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:48.677239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.677201 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:48.771943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.771892 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:48.777526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.777493 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml"] Oct 30 03:35:48.782818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.782798 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:48.798808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.798788 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml"] Oct 30 03:35:48.806486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.806461 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-817531\"" Oct 30 03:35:48.806631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.806506 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-817531-dockercfg-qh4g8\"" Oct 30 03:35:48.863998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.863945 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:48.875605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.875575 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84kr\" (UniqueName: \"kubernetes.io/projected/2c775e5d-3134-40d3-8d81-4748781896a5-kube-api-access-m84kr\") pod \"payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:48.875816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.875713 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c775e5d-3134-40d3-8d81-4748781896a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:48.988127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.987658 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c775e5d-3134-40d3-8d81-4748781896a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:48.988127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.976702 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c775e5d-3134-40d3-8d81-4748781896a5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:48.988127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:48.987895 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84kr\" (UniqueName: \"kubernetes.io/projected/2c775e5d-3134-40d3-8d81-4748781896a5-kube-api-access-m84kr\") pod \"payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:49.005689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.005654 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84kr\" (UniqueName: \"kubernetes.io/projected/2c775e5d-3134-40d3-8d81-4748781896a5-kube-api-access-m84kr\") pod \"payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:49.101250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.101213 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:49.410612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.410589 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml"] Oct 30 03:35:49.411586 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:49.411560 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c775e5d_3134_40d3_8d81_4748781896a5.slice/crio-667f2b59cd7ea21cca19cf4cdcb09f91f714940162e7b006840c0610c828736b WatchSource:0}: Error finding container 667f2b59cd7ea21cca19cf4cdcb09f91f714940162e7b006840c0610c828736b: Status 404 returned error can't find the container with id 667f2b59cd7ea21cca19cf4cdcb09f91f714940162e7b006840c0610c828736b Oct 30 03:35:49.587425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.587396 4442 generic.go:356] "Generic (PLEG): container finished" podID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerID="f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff" exitCode=0 Oct 30 03:35:49.587581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.587467 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" event={"ID":"cb022099-cfd2-41b8-aabc-c6cb77f40112","Type":"ContainerDied","Data":"f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff"} Oct 30 03:35:49.587947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.587924 4442 scope.go:122] "RemoveContainer" containerID="f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff" Oct 30 03:35:49.589277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.589254 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" event={"ID":"2c775e5d-3134-40d3-8d81-4748781896a5","Type":"ContainerStarted","Data":"8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072"} Oct 30 03:35:49.589396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.589296 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" event={"ID":"2c775e5d-3134-40d3-8d81-4748781896a5","Type":"ContainerStarted","Data":"667f2b59cd7ea21cca19cf4cdcb09f91f714940162e7b006840c0610c828736b"} Oct 30 03:35:49.589396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.589321 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:35:49.589511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.589457 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:35:49.589572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.589543 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:35:49.589629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.589596 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:35:49.664573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:49.664515 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" podStartSLOduration=1.664496215 podStartE2EDuration="1.664496215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:49.660693003 +0000 UTC m=+1973.663046202" watchObservedRunningTime="2026-10-30 03:35:49.664496215 +0000 UTC m=+1973.666849412" Oct 30 03:35:50.161903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:50.161861 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:50.473071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:50.472985 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:50.613398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:50.613360 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" event={"ID":"cb022099-cfd2-41b8-aabc-c6cb77f40112","Type":"ContainerStarted","Data":"330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8"} Oct 30 03:35:51.173772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:51.173719 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:51.174869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:51.174835 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 03:35:51.175024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:51.174886 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 03:35:52.108219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:52.108185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:52.175186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:52.175139 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 03:35:52.175386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:52.175245 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 03:35:52.511594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:52.511553 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:52.534904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:52.534877 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 03:35:52.535088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:52.534960 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 03:35:52.612268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:52.612217 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:53.173575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:53.173539 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 03:35:53.173985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:53.173594 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 03:35:53.723582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:53.723538 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:54.174074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:54.174030 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 03:35:54.174473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:54.174104 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 03:35:54.617686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:54.617629 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:54.874089 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:35:54.874007 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:35:55.173251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:55.173180 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 03:35:55.173251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:55.173242 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 03:35:55.299881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:55.299845 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:55.418699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:55.418661 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:56.177958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.177888 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:56.178892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.178874 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 03:35:56.790112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.790080 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8"] Oct 30 03:35:56.794338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.794321 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.807249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.807225 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-852107-serving-cert\"" Oct 30 03:35:56.821669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.821635 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8"] Oct 30 03:35:56.871569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.871527 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:56.882930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.882906 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqj7q\" (UniqueName: \"kubernetes.io/projected/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-kube-api-access-pqj7q\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.883057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.882943 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.883057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.883048 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.883164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.883074 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.973087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.973054 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:56.984814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.984787 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqj7q\" (UniqueName: \"kubernetes.io/projected/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-kube-api-access-pqj7q\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.984946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.984822 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.984946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.984873 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.984946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.984892 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.985601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.985576 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.987680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.987657 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:56.987796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:56.987688 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:57.002076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:57.002049 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqj7q\" (UniqueName: \"kubernetes.io/projected/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-kube-api-access-pqj7q\") pod \"maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:57.109861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:57.109774 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:57.192118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:57.192072 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:57.407074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:57.407045 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8"] Oct 30 03:35:57.408160 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:35:57.408135 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86d4db2_5c40_4ff1_a390_6dbf6c8204e1.slice/crio-985b9c22c09b7f84514144a42708b693d00c94b9503ad95a694d6c938c50b677 WatchSource:0}: Error finding container 985b9c22c09b7f84514144a42708b693d00c94b9503ad95a694d6c938c50b677: Status 404 returned error can't find the container with id 985b9c22c09b7f84514144a42708b693d00c94b9503ad95a694d6c938c50b677 Oct 30 03:35:57.462080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:57.462045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:57.680520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:57.680427 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" event={"ID":"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1","Type":"ContainerStarted","Data":"985b9c22c09b7f84514144a42708b693d00c94b9503ad95a694d6c938c50b677"} Oct 30 03:35:58.495029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:58.494996 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:35:58.689156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:58.689120 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" event={"ID":"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1","Type":"ContainerStarted","Data":"2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338"} Oct 30 03:35:58.689369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:58.689192 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:35:58.729835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:58.729779 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" podStartSLOduration=2.402752198 podStartE2EDuration="2.729742977s" totalImagesPullingTime="326.990779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:58.728453906 +0000 UTC m=+1982.730807104" watchObservedRunningTime="2026-10-30 03:35:58.729742977 +0000 UTC m=+1982.732096174" Oct 30 03:35:59.966062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:35:59.966029 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:00.616920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:00.616892 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 03:36:01.368520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:01.368491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:02.861682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:02.861653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:04.171148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:04.171090 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:04.700637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:04.700609 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 03:36:05.620307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:05.620276 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:05.757284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:05.752428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:05.866265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:05.866223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:07.216119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:07.216075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:08.687242 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:08.687209 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:10.190846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:10.190813 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:11.097251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:11.097214 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:36:11.670049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:11.670014 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:13.207567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:13.207532 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:14.689647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:14.689615 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:16.065439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:16.065405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:16.643544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:16.643501 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:17.707333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:17.707262 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:19.314591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:19.314539 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:20.790059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:20.790006 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:22.254901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:22.254865 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:23.577560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:23.577528 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:24.143188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:24.143148 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:24.861302 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:36:24.861262 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:36:25.005729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:25.005694 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:26.663509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:26.663478 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:28.183517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:28.183484 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:29.791635 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:29.791599 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:31.301424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:31.301391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:31.479624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:31.479591 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:32.312555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:32.312521 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:32.892283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:32.892248 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:34.440090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:34.440060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:35.814050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:35.814013 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:37.221379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:37.221347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:38.258057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.258027 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7"] Oct 30 03:36:38.263144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.263123 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.276488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.276469 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-1a1b45-openshift-default-dockercfg-gx88g\"" Oct 30 03:36:38.291718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.291690 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7"] Oct 30 03:36:38.374937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.374906 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a5b40-7745-480e-b79c-69ea11956806-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.374944 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfjn\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-kube-api-access-4tfjn\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.374970 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.375051 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.375105 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.375150 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a5b40-7745-480e-b79c-69ea11956806-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.375238 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.375306 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.375354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.375331 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476096 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476202 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476228 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a5b40-7745-480e-b79c-69ea11956806-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476260 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfjn\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-kube-api-access-4tfjn\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476313 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476297 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476331 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476384 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476438 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a5b40-7745-480e-b79c-69ea11956806-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476579 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476721 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476865 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.476955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.476937 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.477061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.477038 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a5b40-7745-480e-b79c-69ea11956806-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.479172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.479138 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.479608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.479586 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a5b40-7745-480e-b79c-69ea11956806-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.494104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.494075 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.494195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.494169 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfjn\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-kube-api-access-4tfjn\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.577107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.577048 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:38.621224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.621189 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:38.889695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.889668 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7"] Oct 30 03:36:38.890489 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:36:38.890457 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6a5b40_7745_480e_b79c_69ea11956806.slice/crio-7f5dc2113a3b94abfa9f1541ecee9e6f7487c89857bb14ce5ffb98490d07b201 WatchSource:0}: Error finding container 7f5dc2113a3b94abfa9f1541ecee9e6f7487c89857bb14ce5ffb98490d07b201: Status 404 returned error can't find the container with id 7f5dc2113a3b94abfa9f1541ecee9e6f7487c89857bb14ce5ffb98490d07b201 Oct 30 03:36:38.892978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.892949 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:36:38.893051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.893015 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:36:38.893051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.893041 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:36:38.981497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.981461 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" event={"ID":"7c6a5b40-7745-480e-b79c-69ea11956806","Type":"ContainerStarted","Data":"e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7"} Oct 30 03:36:38.981657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:38.981527 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" event={"ID":"7c6a5b40-7745-480e-b79c-69ea11956806","Type":"ContainerStarted","Data":"7f5dc2113a3b94abfa9f1541ecee9e6f7487c89857bb14ce5ffb98490d07b201"} Oct 30 03:36:39.023089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:39.023030 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podStartSLOduration=1.023011401 podStartE2EDuration="1.023011401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:39.019405519 +0000 UTC m=+2023.021758716" watchObservedRunningTime="2026-10-30 03:36:39.023011401 +0000 UTC m=+2023.025364599" Oct 30 03:36:39.578032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:39.577998 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:39.579524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:39.579497 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 03:36:39.579654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:39.579545 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 03:36:40.242697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:40.242663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:40.577590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:40.577556 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 03:36:40.577816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:40.577605 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 03:36:41.071279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:41.071242 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:41.577952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:41.577914 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 03:36:41.578125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:41.577975 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 03:36:41.943934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:41.943838 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:42.578291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:42.578259 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 03:36:42.578677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:42.578309 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 03:36:43.434191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:43.434149 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:43.578374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:43.578337 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 03:36:43.578783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:43.578402 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 03:36:44.578332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:44.578296 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 03:36:44.578513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:44.578356 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 03:36:45.046951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:45.046902 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:45.582122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:45.582091 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:45.582590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:45.582489 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:45.583222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:45.583205 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:36:46.499767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:46.499722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:47.278694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:47.278661 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:47.900318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:47.900290 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:48.321323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.321289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:48.574097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.574012 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2"] Oct 30 03:36:48.578132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.578110 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.592371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.592342 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6ad168\"/\"default-dockercfg-p6fdw\"" Oct 30 03:36:48.596775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.596726 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2"] Oct 30 03:36:48.603364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.603342 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6ad168\"/\"openshift-service-ca.crt\"" Oct 30 03:36:48.603519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.603373 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6ad168\"/\"auth-test-model-6ad168-kserve-self-signed-certs\"" Oct 30 03:36:48.603604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.603473 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6ad168\"/\"kube-root-ca.crt\"" Oct 30 03:36:48.615005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.614974 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-dshm\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.615124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.615016 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-home\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.615187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.615153 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tls-certs\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.615261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.615241 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-model-cache\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.615325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.615270 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tmp-dir\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.615406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.615386 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-kube-api-access-rb5df\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.621741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.621720 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:48.716090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716052 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-dshm\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716108 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-home\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716176 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tls-certs\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716211 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-model-cache\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716234 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tmp-dir\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716279 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-kube-api-access-rb5df\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716565 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-home\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716635 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-model-cache\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.716701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.716663 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tmp-dir\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.719128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.719106 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-dshm\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.719331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.719312 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tls-certs\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.733941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.733915 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-kube-api-access-rb5df\") pod \"auth-test-model-6ad168-kserve-dc644cfd9-n6pb2\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:48.897505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:48.897412 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:49.193407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:49.193329 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2"] Oct 30 03:36:49.198199 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:36:49.198168 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9efaf3a_5f36_4d00_bf14_a0a052e7ac7e.slice/crio-32809a7c3457d24c381ff99fe4fcea68417be0fc4157910a75424367c8eb5932 WatchSource:0}: Error finding container 32809a7c3457d24c381ff99fe4fcea68417be0fc4157910a75424367c8eb5932: Status 404 returned error can't find the container with id 32809a7c3457d24c381ff99fe4fcea68417be0fc4157910a75424367c8eb5932 Oct 30 03:36:49.492734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:49.492662 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:50.069588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:50.069552 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" event={"ID":"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e","Type":"ContainerStarted","Data":"ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f"} Oct 30 03:36:50.069794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:50.069601 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" event={"ID":"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e","Type":"ContainerStarted","Data":"32809a7c3457d24c381ff99fe4fcea68417be0fc4157910a75424367c8eb5932"} Oct 30 03:36:50.106419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:50.106370 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" podStartSLOduration=2.106356212 podStartE2EDuration="2.106356212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:36:50.104358568 +0000 UTC m=+2034.106711766" watchObservedRunningTime="2026-10-30 03:36:50.106356212 +0000 UTC m=+2034.108709409" Oct 30 03:36:50.277441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:50.277405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:50.675421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:50.675385 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:36:50.897219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:50.897185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:50.898105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:50.898089 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:52.592694 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:52.592645 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:54.089570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:54.089541 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:54.860619 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:36:54.860570 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:36:55.628517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:55.628485 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:57.085342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:57.085311 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:58.597377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:58.597341 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:36:58.898571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:58.898485 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:58.911504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:58.911473 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:36:59.137157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:36:59.137128 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:37:00.166052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:00.166023 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:01.709684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:01.709650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:01.812150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:01.812121 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:03.185047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:03.185010 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:04.667871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:04.667837 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:06.204312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:06.204272 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:07.610635 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:07.610601 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:09.257282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:09.257248 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:10.726237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:10.726195 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:12.217074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:12.217042 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:13.817455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:13.817425 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:15.286939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:15.286901 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:16.781866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:16.781824 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:18.156609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:18.156578 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:19.594353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:19.594320 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:20.981769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:20.981706 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:22.495126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:22.495096 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:22.583683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:22.583654 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:23.834894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.834859 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw"] Oct 30 03:37:23.838960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.838937 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.853043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.853023 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7d99a4-openshift-default-dockercfg-wfkbf\"" Oct 30 03:37:23.866816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.866793 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw"] Oct 30 03:37:23.896680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.896640 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966592bb-0715-43e7-aa0e-01f043c68794-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.896680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.896691 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966592bb-0715-43e7-aa0e-01f043c68794-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.896914 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.896867 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.896964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.896922 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.897002 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.896991 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.897060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.897044 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.897136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.897124 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.897176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.897160 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn29g\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-kube-api-access-kn29g\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.897219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.897190 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.981026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.980991 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:23.998439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998414 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998452 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998495 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998565 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998624 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998664 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn29g\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-kube-api-access-kn29g\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998711 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998743 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966592bb-0715-43e7-aa0e-01f043c68794-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.998836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998803 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966592bb-0715-43e7-aa0e-01f043c68794-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.999088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998924 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.999088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.998960 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.999304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.999279 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.999429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.999393 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966592bb-0715-43e7-aa0e-01f043c68794-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:23.999492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:23.999430 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:24.001482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.001441 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:24.001634 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.001617 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966592bb-0715-43e7-aa0e-01f043c68794-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:24.018508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.018481 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:24.018632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.018546 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn29g\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-kube-api-access-kn29g\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:24.152240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.152155 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:24.467098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.467070 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw"] Oct 30 03:37:24.468821 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:37:24.468792 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966592bb_0715_43e7_aa0e_01f043c68794.slice/crio-fc40919a0e925ffdedfca48e6293729e95d5b5f2364b229ca2ec664e991d6f33 WatchSource:0}: Error finding container fc40919a0e925ffdedfca48e6293729e95d5b5f2364b229ca2ec664e991d6f33: Status 404 returned error can't find the container with id fc40919a0e925ffdedfca48e6293729e95d5b5f2364b229ca2ec664e991d6f33 Oct 30 03:37:24.471430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.471396 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:37:24.471526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.471469 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:37:24.471526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:24.471511 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:37:24.861691 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:37:24.861648 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:37:25.314720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:25.314682 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" event={"ID":"966592bb-0715-43e7-aa0e-01f043c68794","Type":"ContainerStarted","Data":"d9abb2e2c4cd416e5f29332be9cac5a7e7c1c178db5577b38e39bf8bee688797"} Oct 30 03:37:25.314955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:25.314736 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" event={"ID":"966592bb-0715-43e7-aa0e-01f043c68794","Type":"ContainerStarted","Data":"fc40919a0e925ffdedfca48e6293729e95d5b5f2364b229ca2ec664e991d6f33"} Oct 30 03:37:25.365296 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:25.365245 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" podStartSLOduration=2.365231011 podStartE2EDuration="2.365231011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:37:25.363195207 +0000 UTC m=+2069.365548405" watchObservedRunningTime="2026-10-30 03:37:25.365231011 +0000 UTC m=+2069.367584208" Oct 30 03:37:25.609532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:25.609416 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:26.152824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:26.152789 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:26.154577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:26.154555 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 03:37:26.154672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:26.154630 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 03:37:27.152466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:27.152425 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 03:37:27.152674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:27.152484 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 03:37:27.224117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:27.224087 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:28.153375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:28.153339 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 03:37:28.153598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:28.153394 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 03:37:29.026747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:29.026719 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:29.153251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:29.153218 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 03:37:29.153431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:29.153268 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 03:37:30.153158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:30.153125 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 03:37:30.153570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:30.153176 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 03:37:30.528539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:30.528508 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:31.156833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:31.156806 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:31.157298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:31.157072 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:31.157825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:31.157808 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:37:32.078467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:32.078435 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:33.522217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:33.522183 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:33.796473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:33.796351 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:34.301061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:34.301033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:34.425007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:34.420068 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:34.514840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:34.514805 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:37:35.074814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:35.074778 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:36.730315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:36.730265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:37.418170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:37.418131 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:37.520454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:37.520418 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:37.643299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:37.643266 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:37.743514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:37.743422 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:38.262211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:38.262161 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:39.165684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:39.165636 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:39.263748 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:39.263718 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:40.005804 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:40.005743 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:41.572446 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:41.572409 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:42.980965 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:42.980937 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:44.410311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:44.410259 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:45.942916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:45.942884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:47.512315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:47.512283 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:48.411678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:48.411638 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:48.512782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:48.512735 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:48.896979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:48.896949 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:50.392589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:50.392561 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:50.492985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:50.492949 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:51.798346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:51.798311 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:53.200258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:53.200227 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:54.594442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:54.594409 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:54.861018 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:37:54.860933 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:37:55.765954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:55.765916 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:55.861542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:55.861511 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:56.103698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:56.103612 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:56.419328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:56.419253 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:37:56.426043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:56.426015 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:37:57.569959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:57.569911 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:57.667962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:57.667906 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:57.774217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:57.774179 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:37:59.305832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:37:59.305793 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:00.904086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:00.904050 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:02.428954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:02.428911 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:03.790113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:03.790069 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:05.245580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:05.245547 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:06.793502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:06.793474 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:08.272947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:08.272912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:09.849522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:09.849407 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:11.266647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:11.266617 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:12.725214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:12.725164 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:12.938544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:12.938506 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:38:14.306940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:14.306907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:15.801257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:15.801223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:17.286566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:17.286524 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:18.721813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:18.721780 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:20.288436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:20.288401 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:21.887938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:21.887899 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:23.127719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:23.127684 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:23.302722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:23.302685 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:24.812499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:24.812467 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:24.861432 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:38:24.861386 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:38:25.694722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.694691 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw"] Oct 30 03:38:25.699520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.699494 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.725610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.725582 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw"] Oct 30 03:38:25.733489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.733462 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-75c2b0\"/\"openshift-service-ca.crt\"" Oct 30 03:38:25.733641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.733543 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-75c2b0\"/\"kube-root-ca.crt\"" Oct 30 03:38:25.733826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.733806 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-75c2b0\"/\"explicit-model-75c2b0-kserve-self-signed-certs\"" Oct 30 03:38:25.733920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.733878 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-75c2b0\"/\"default-dockercfg-vzx2w\"" Oct 30 03:38:25.746066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.746044 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:25.836432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.836401 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6s2q\" (UniqueName: \"kubernetes.io/projected/3dd50d74-fc60-41d8-a086-409d6b173944-kube-api-access-v6s2q\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.836883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.836441 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-model-cache\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.836883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.836466 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-dshm\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.836883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.836606 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd50d74-fc60-41d8-a086-409d6b173944-tls-certs\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.836883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.836637 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-home\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.836883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.836713 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-tmp-dir\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.937823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.937786 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6s2q\" (UniqueName: \"kubernetes.io/projected/3dd50d74-fc60-41d8-a086-409d6b173944-kube-api-access-v6s2q\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.937833 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-model-cache\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.937871 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-dshm\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.937939 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd50d74-fc60-41d8-a086-409d6b173944-tls-certs\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.937965 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-home\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.938033 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-tmp-dir\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.938346 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-model-cache\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.938378 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-tmp-dir\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.938420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.938399 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-home\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.940345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.940317 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-dshm\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.940593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.940574 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd50d74-fc60-41d8-a086-409d6b173944-tls-certs\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:25.999337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:25.999288 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6s2q\" (UniqueName: \"kubernetes.io/projected/3dd50d74-fc60-41d8-a086-409d6b173944-kube-api-access-v6s2q\") pod \"explicit-model-75c2b0-kserve-577bd49865-9lksw\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:26.015344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:26.015307 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:26.320719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:26.320692 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw"] Oct 30 03:38:26.322098 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:38:26.322067 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dd50d74_fc60_41d8_a086_409d6b173944.slice/crio-085b4741c8e28320905a0f840fa5260186918a85a0187fbf2a6da289d31c4907 WatchSource:0}: Error finding container 085b4741c8e28320905a0f840fa5260186918a85a0187fbf2a6da289d31c4907: Status 404 returned error can't find the container with id 085b4741c8e28320905a0f840fa5260186918a85a0187fbf2a6da289d31c4907 Oct 30 03:38:26.414941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:26.414908 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:26.791410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:26.791373 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" event={"ID":"3dd50d74-fc60-41d8-a086-409d6b173944","Type":"ContainerStarted","Data":"90d12de2cd0117d755f143aa33968266845e008c2b9c6e50a14e5adc8160daf2"} Oct 30 03:38:26.791410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:26.791412 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" event={"ID":"3dd50d74-fc60-41d8-a086-409d6b173944","Type":"ContainerStarted","Data":"085b4741c8e28320905a0f840fa5260186918a85a0187fbf2a6da289d31c4907"} Oct 30 03:38:26.834663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:26.834607 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" podStartSLOduration=1.83459293 podStartE2EDuration="1.83459293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:38:26.830993475 +0000 UTC m=+2130.833346673" watchObservedRunningTime="2026-10-30 03:38:26.83459293 +0000 UTC m=+2130.836946160" Oct 30 03:38:27.015917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:27.015887 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:27.926442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:27.926414 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:29.468909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:29.468872 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:30.906512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:30.906478 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:31.669683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:31.669631 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw"] Oct 30 03:38:31.770137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:31.770107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:32.206434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:32.206357 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:32.536430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:32.536400 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:33.990312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:33.990276 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:35.554991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:35.554956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:36.985013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:36.984980 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:37.343151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:37.343108 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:38.414079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:38.414052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:40.005543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:40.005514 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:41.549643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:41.549607 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:43.081274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:43.081242 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:44.485591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:44.485560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:45.935994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:45.935961 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:46.575260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:46.575229 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:46.674958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:46.674925 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:46.692076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:46.692050 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" podUID="3dd50d74-fc60-41d8-a086-409d6b173944" containerName="main" containerID="cri-o://90d12de2cd0117d755f143aa33968266845e008c2b9c6e50a14e5adc8160daf2" gracePeriod=45 Oct 30 03:38:46.931243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:46.931212 4442 generic.go:356] "Generic (PLEG): container finished" podID="3dd50d74-fc60-41d8-a086-409d6b173944" containerID="90d12de2cd0117d755f143aa33968266845e008c2b9c6e50a14e5adc8160daf2" exitCode=0 Oct 30 03:38:46.931432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:46.931283 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" event={"ID":"3dd50d74-fc60-41d8-a086-409d6b173944","Type":"ContainerDied","Data":"90d12de2cd0117d755f143aa33968266845e008c2b9c6e50a14e5adc8160daf2"} Oct 30 03:38:46.981896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:46.981867 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:47.010721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.010691 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dd50d74-fc60-41d8-a086-409d6b173944-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd50d74-fc60-41d8-a086-409d6b173944-tls-certs\") pod \"3dd50d74-fc60-41d8-a086-409d6b173944\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " Oct 30 03:38:47.010857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.010778 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd50d74-fc60-41d8-a086-409d6b173944-kube-api-access-v6s2q\" (UniqueName: \"kubernetes.io/projected/3dd50d74-fc60-41d8-a086-409d6b173944-kube-api-access-v6s2q\") pod \"3dd50d74-fc60-41d8-a086-409d6b173944\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " Oct 30 03:38:47.010857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.010828 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-tmp-dir\") pod \"3dd50d74-fc60-41d8-a086-409d6b173944\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " Oct 30 03:38:47.010975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.010873 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-model-cache\") pod \"3dd50d74-fc60-41d8-a086-409d6b173944\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " Oct 30 03:38:47.011036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.011014 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-dshm\") pod \"3dd50d74-fc60-41d8-a086-409d6b173944\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " Oct 30 03:38:47.011098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.011050 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-home\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-home\") pod \"3dd50d74-fc60-41d8-a086-409d6b173944\" (UID: \"3dd50d74-fc60-41d8-a086-409d6b173944\") " Oct 30 03:38:47.011175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.011151 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-tmp-dir" pod "3dd50d74-fc60-41d8-a086-409d6b173944" (UID: "3dd50d74-fc60-41d8-a086-409d6b173944"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:38:47.011434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.011368 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-home" pod "3dd50d74-fc60-41d8-a086-409d6b173944" (UID: "3dd50d74-fc60-41d8-a086-409d6b173944"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:38:47.011434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.011388 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-model-cache" pod "3dd50d74-fc60-41d8-a086-409d6b173944" (UID: "3dd50d74-fc60-41d8-a086-409d6b173944"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:38:47.011720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.011458 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:38:47.011720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.011591 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:38:47.013187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.013160 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd50d74-fc60-41d8-a086-409d6b173944-tls-certs" pod "3dd50d74-fc60-41d8-a086-409d6b173944" (UID: "3dd50d74-fc60-41d8-a086-409d6b173944"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:38:47.013273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.013248 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd50d74-fc60-41d8-a086-409d6b173944-kube-api-access-v6s2q" pod "3dd50d74-fc60-41d8-a086-409d6b173944" (UID: "3dd50d74-fc60-41d8-a086-409d6b173944"). InnerVolumeSpecName "kube-api-access-v6s2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:38:47.013379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.013357 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-dshm" pod "3dd50d74-fc60-41d8-a086-409d6b173944" (UID: "3dd50d74-fc60-41d8-a086-409d6b173944"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:38:47.113145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.113065 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:38:47.113366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.113349 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd50d74-fc60-41d8-a086-409d6b173944-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:38:47.113479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.113462 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd50d74-fc60-41d8-a086-409d6b173944-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:38:47.113541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.113482 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6s2q\" (UniqueName: \"kubernetes.io/projected/3dd50d74-fc60-41d8-a086-409d6b173944-kube-api-access-v6s2q\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:38:47.396916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.396837 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:47.939071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.939040 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" Oct 30 03:38:47.939273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.939087 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw" event={"ID":"3dd50d74-fc60-41d8-a086-409d6b173944","Type":"ContainerDied","Data":"085b4741c8e28320905a0f840fa5260186918a85a0187fbf2a6da289d31c4907"} Oct 30 03:38:47.939273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.939142 4442 scope.go:122] "RemoveContainer" containerID="90d12de2cd0117d755f143aa33968266845e008c2b9c6e50a14e5adc8160daf2" Oct 30 03:38:47.987166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.987135 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw"] Oct 30 03:38:47.996661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:47.996639 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-75c2b0/explicit-model-75c2b0-kserve-577bd49865-9lksw"] Oct 30 03:38:48.106856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:48.106829 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd50d74-fc60-41d8-a086-409d6b173944" path="/var/lib/kubelet/pods/3dd50d74-fc60-41d8-a086-409d6b173944/volumes" Oct 30 03:38:48.545928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:48.545896 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:49.033539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:49.033500 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:49.562011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:49.561983 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:50.503321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:50.503280 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:50.603004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:50.602972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:52.188406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:52.188371 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:53.803809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:53.803775 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:54.859852 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:38:54.859807 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:38:55.296781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:55.296729 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:56.783522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:56.783490 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:58.314151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:58.314112 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:38:59.808774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:38:59.808710 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:01.196063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:01.196032 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:02.425945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:02.425912 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:39:02.746871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:02.746791 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:04.287517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:04.287483 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:04.664854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:04.664770 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:05.098506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:05.098468 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:05.778311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:05.778277 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:07.169426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:07.169394 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:08.083818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.083786 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:08.351145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.351064 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-ad8907"] Oct 30 03:39:08.351617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.351605 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3dd50d74-fc60-41d8-a086-409d6b173944" containerName="main" Oct 30 03:39:08.351660 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.351618 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd50d74-fc60-41d8-a086-409d6b173944" containerName="main" Oct 30 03:39:08.351734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.351725 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd50d74-fc60-41d8-a086-409d6b173944" containerName="main" Oct 30 03:39:08.356709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.356686 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:08.368855 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.368831 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-ad8907"] Oct 30 03:39:08.376220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.376201 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jmnx7\"" Oct 30 03:39:08.465187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.465151 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h592s\" (UniqueName: \"kubernetes.io/projected/882517c0-5f4a-4421-a4d8-ba89a9a2ac91-kube-api-access-h592s\") pod \"test-curl-7579-ad8907\" (UID: \"882517c0-5f4a-4421-a4d8-ba89a9a2ac91\") " pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:08.566152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.566115 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h592s\" (UniqueName: \"kubernetes.io/projected/882517c0-5f4a-4421-a4d8-ba89a9a2ac91-kube-api-access-h592s\") pod \"test-curl-7579-ad8907\" (UID: \"882517c0-5f4a-4421-a4d8-ba89a9a2ac91\") " pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:08.586038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.586002 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h592s\" (UniqueName: \"kubernetes.io/projected/882517c0-5f4a-4421-a4d8-ba89a9a2ac91-kube-api-access-h592s\") pod \"test-curl-7579-ad8907\" (UID: \"882517c0-5f4a-4421-a4d8-ba89a9a2ac91\") " pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:08.669985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.669901 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:08.704392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.704366 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:08.967685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:08.967660 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-ad8907"] Oct 30 03:39:08.968550 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:39:08.968526 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod882517c0_5f4a_4421_a4d8_ba89a9a2ac91.slice/crio-492cc56f9f63479081bda1790bad985942d3e4760e9b3e14d4af7990ebf256ac WatchSource:0}: Error finding container 492cc56f9f63479081bda1790bad985942d3e4760e9b3e14d4af7990ebf256ac: Status 404 returned error can't find the container with id 492cc56f9f63479081bda1790bad985942d3e4760e9b3e14d4af7990ebf256ac Oct 30 03:39:09.070668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:09.070635 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-ad8907" event={"ID":"882517c0-5f4a-4421-a4d8-ba89a9a2ac91","Type":"ContainerStarted","Data":"492cc56f9f63479081bda1790bad985942d3e4760e9b3e14d4af7990ebf256ac"} Oct 30 03:39:10.208023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:10.207982 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:11.086270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:11.086234 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-ad8907" event={"ID":"882517c0-5f4a-4421-a4d8-ba89a9a2ac91","Type":"ContainerStarted","Data":"6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2"} Oct 30 03:39:11.123269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:11.123219 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-ad8907" podStartSLOduration=1.218494767 podStartE2EDuration="3.123205024s" totalImagesPullingTime="1.904710257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:11.120453799 +0000 UTC m=+2175.122806996" watchObservedRunningTime="2026-10-30 03:39:11.123205024 +0000 UTC m=+2175.125558220" Oct 30 03:39:11.789691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:11.789660 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:12.013448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.013414 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-ad8907"] Oct 30 03:39:12.020153 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.020125 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-ad8907"] Oct 30 03:39:12.222304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.222223 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-a28246"] Oct 30 03:39:12.222791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.222778 4442 cpu_manager.go:414] "containerMap: removing container" podUID="882517c0-5f4a-4421-a4d8-ba89a9a2ac91" containerName="test-curl-7579-ad8907" Oct 30 03:39:12.222855 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.222793 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882517c0-5f4a-4421-a4d8-ba89a9a2ac91" containerName="test-curl-7579-ad8907" Oct 30 03:39:12.222895 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.222869 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="882517c0-5f4a-4421-a4d8-ba89a9a2ac91" containerName="test-curl-7579-ad8907" Oct 30 03:39:12.226417 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.226396 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-a28246" Oct 30 03:39:12.242307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.242285 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-a28246"] Oct 30 03:39:12.331173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.331138 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29w5r\" (UniqueName: \"kubernetes.io/projected/2ae98857-42d4-4164-b468-309d4a6accfe-kube-api-access-29w5r\") pod \"test-curl-7579-a28246\" (UID: \"2ae98857-42d4-4164-b468-309d4a6accfe\") " pod="opendatahub/test-curl-7579-a28246" Oct 30 03:39:12.432364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.432327 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29w5r\" (UniqueName: \"kubernetes.io/projected/2ae98857-42d4-4164-b468-309d4a6accfe-kube-api-access-29w5r\") pod \"test-curl-7579-a28246\" (UID: \"2ae98857-42d4-4164-b468-309d4a6accfe\") " pod="opendatahub/test-curl-7579-a28246" Oct 30 03:39:12.452914 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.452885 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29w5r\" (UniqueName: \"kubernetes.io/projected/2ae98857-42d4-4164-b468-309d4a6accfe-kube-api-access-29w5r\") pod \"test-curl-7579-a28246\" (UID: \"2ae98857-42d4-4164-b468-309d4a6accfe\") " pod="opendatahub/test-curl-7579-a28246" Oct 30 03:39:12.538952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.538923 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-a28246" Oct 30 03:39:12.832988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:12.832960 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-a28246"] Oct 30 03:39:12.834359 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:39:12.834332 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ae98857_42d4_4164_b468_309d4a6accfe.slice/crio-1c62efe060ae451a32523c0985cba8ae35223650b9fd3eb08525be585cea5961 WatchSource:0}: Error finding container 1c62efe060ae451a32523c0985cba8ae35223650b9fd3eb08525be585cea5961: Status 404 returned error can't find the container with id 1c62efe060ae451a32523c0985cba8ae35223650b9fd3eb08525be585cea5961 Oct 30 03:39:13.100011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:13.099917 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-a28246" event={"ID":"2ae98857-42d4-4164-b468-309d4a6accfe","Type":"ContainerStarted","Data":"73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40"} Oct 30 03:39:13.100011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:13.099962 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-a28246" event={"ID":"2ae98857-42d4-4164-b468-309d4a6accfe","Type":"ContainerStarted","Data":"1c62efe060ae451a32523c0985cba8ae35223650b9fd3eb08525be585cea5961"} Oct 30 03:39:13.100429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:13.100002 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-ad8907" podUID="882517c0-5f4a-4421-a4d8-ba89a9a2ac91" containerName="test-curl-7579-ad8907" containerID="cri-o://6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2" gracePeriod=2 Oct 30 03:39:13.133424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:13.133372 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-a28246" podStartSLOduration=1.133358985 podStartE2EDuration="1.133358985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:13.132738468 +0000 UTC m=+2177.135091670" watchObservedRunningTime="2026-10-30 03:39:13.133358985 +0000 UTC m=+2177.135712229" Oct 30 03:39:13.211580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:13.211552 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:13.952673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:13.952634 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-a28246"] Oct 30 03:39:13.956598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:13.956565 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-a28246"] Oct 30 03:39:14.111210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.111181 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-18c4cb"] Oct 30 03:39:14.111671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.111651 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2ae98857-42d4-4164-b468-309d4a6accfe" containerName="test-curl-7579-a28246" Oct 30 03:39:14.111671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.111666 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae98857-42d4-4164-b468-309d4a6accfe" containerName="test-curl-7579-a28246" Oct 30 03:39:14.111843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.111822 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae98857-42d4-4164-b468-309d4a6accfe" containerName="test-curl-7579-a28246" Oct 30 03:39:14.115309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.115293 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-18c4cb" Oct 30 03:39:14.124279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.124259 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-18c4cb"] Oct 30 03:39:14.153281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.153247 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hj6h\" (UniqueName: \"kubernetes.io/projected/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd-kube-api-access-7hj6h\") pod \"test-curl-7579-18c4cb\" (UID: \"d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd\") " pod="opendatahub/test-curl-7579-18c4cb" Oct 30 03:39:14.254536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.254507 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hj6h\" (UniqueName: \"kubernetes.io/projected/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd-kube-api-access-7hj6h\") pod \"test-curl-7579-18c4cb\" (UID: \"d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd\") " pod="opendatahub/test-curl-7579-18c4cb" Oct 30 03:39:14.272142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.272115 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hj6h\" (UniqueName: \"kubernetes.io/projected/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd-kube-api-access-7hj6h\") pod \"test-curl-7579-18c4cb\" (UID: \"d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd\") " pod="opendatahub/test-curl-7579-18c4cb" Oct 30 03:39:14.429898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.429868 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-18c4cb" Oct 30 03:39:14.510630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.510547 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:14.729705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.729666 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-18c4cb"] Oct 30 03:39:14.733796 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:39:14.733731 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7bb834a_50ff_4ebb_aeb5_a2d76f1039bd.slice/crio-9d6bc3fd217fd5f310ebbe6d28e911a8bc88290101fd27a3b225ab63074a114b WatchSource:0}: Error finding container 9d6bc3fd217fd5f310ebbe6d28e911a8bc88290101fd27a3b225ab63074a114b: Status 404 returned error can't find the container with id 9d6bc3fd217fd5f310ebbe6d28e911a8bc88290101fd27a3b225ab63074a114b Oct 30 03:39:14.819362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.819339 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:14.929510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:14.929468 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:15.058451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.058338 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:15.116742 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.116709 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-18c4cb" event={"ID":"d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd","Type":"ContainerStarted","Data":"6304ed49c1748375cbe1f86d09d629426033f79cbaa0a435737593ae8a24fbe5"} Oct 30 03:39:15.116896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.116774 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-18c4cb" event={"ID":"d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd","Type":"ContainerStarted","Data":"9d6bc3fd217fd5f310ebbe6d28e911a8bc88290101fd27a3b225ab63074a114b"} Oct 30 03:39:15.116960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.116939 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-a28246" podUID="2ae98857-42d4-4164-b468-309d4a6accfe" containerName="test-curl-7579-a28246" containerID="cri-o://73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40" gracePeriod=2 Oct 30 03:39:15.153650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.153606 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-18c4cb" podStartSLOduration=1.153593446 podStartE2EDuration="1.153593446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:15.150771142 +0000 UTC m=+2179.153124340" watchObservedRunningTime="2026-10-30 03:39:15.153593446 +0000 UTC m=+2179.155946637" Oct 30 03:39:15.300142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.300117 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:15.366926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.366860 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882517c0-5f4a-4421-a4d8-ba89a9a2ac91-kube-api-access-h592s\" (UniqueName: \"kubernetes.io/projected/882517c0-5f4a-4421-a4d8-ba89a9a2ac91-kube-api-access-h592s\") pod \"882517c0-5f4a-4421-a4d8-ba89a9a2ac91\" (UID: \"882517c0-5f4a-4421-a4d8-ba89a9a2ac91\") " Oct 30 03:39:15.369148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.369123 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882517c0-5f4a-4421-a4d8-ba89a9a2ac91-kube-api-access-h592s" pod "882517c0-5f4a-4421-a4d8-ba89a9a2ac91" (UID: "882517c0-5f4a-4421-a4d8-ba89a9a2ac91"). InnerVolumeSpecName "kube-api-access-h592s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:15.468079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:15.468042 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h592s\" (UniqueName: \"kubernetes.io/projected/882517c0-5f4a-4421-a4d8-ba89a9a2ac91-kube-api-access-h592s\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:39:16.066408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.066372 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-18c4cb"] Oct 30 03:39:16.069692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.069663 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-18c4cb"] Oct 30 03:39:16.129048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.129001 4442 generic.go:356] "Generic (PLEG): container finished" podID="882517c0-5f4a-4421-a4d8-ba89a9a2ac91" containerID="6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2" exitCode=137 Oct 30 03:39:16.130292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.130265 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:16.143930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.143900 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882517c0-5f4a-4421-a4d8-ba89a9a2ac91" path="/var/lib/kubelet/pods/882517c0-5f4a-4421-a4d8-ba89a9a2ac91/volumes" Oct 30 03:39:16.144429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.144413 4442 scope.go:122] "RemoveContainer" containerID="6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2" Oct 30 03:39:16.153585 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:39:16.153547 4442 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7579-ad8907\" is forbidden: User \"system:node:ip-10-0-134-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-92.ec2.internal' and this object" podUID="882517c0-5f4a-4421-a4d8-ba89a9a2ac91" pod="opendatahub/test-curl-7579-ad8907" Oct 30 03:39:16.154639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.154624 4442 scope.go:122] "RemoveContainer" containerID="6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2" Oct 30 03:39:16.154954 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:39:16.154932 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2\": container with ID starting with 6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2 not found: ID does not exist" containerID="6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2" Oct 30 03:39:16.155058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.154964 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2"} err="failed to get container status \"6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2\": rpc error: code = NotFound desc = could not find container \"6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2\": container with ID starting with 6e3f89451346030d3134cda773919b08c03a489fccb60a650225d585002e20c2 not found: ID does not exist" Oct 30 03:39:16.226382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.226329 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-f96b2a"] Oct 30 03:39:16.227411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.227380 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd" containerName="test-curl-7579-18c4cb" Oct 30 03:39:16.227411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.227405 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd" containerName="test-curl-7579-18c4cb" Oct 30 03:39:16.227607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.227514 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd" containerName="test-curl-7579-18c4cb" Oct 30 03:39:16.232305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.232288 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-f96b2a" Oct 30 03:39:16.245044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.245022 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-f96b2a"] Oct 30 03:39:16.275523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.275497 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnc2\" (UniqueName: \"kubernetes.io/projected/b14f496e-647e-4739-88c8-29f41f17b3ac-kube-api-access-qgnc2\") pod \"test-curl-7579-f96b2a\" (UID: \"b14f496e-647e-4739-88c8-29f41f17b3ac\") " pod="opendatahub/test-curl-7579-f96b2a" Oct 30 03:39:16.376563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.376495 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnc2\" (UniqueName: \"kubernetes.io/projected/b14f496e-647e-4739-88c8-29f41f17b3ac-kube-api-access-qgnc2\") pod \"test-curl-7579-f96b2a\" (UID: \"b14f496e-647e-4739-88c8-29f41f17b3ac\") " pod="opendatahub/test-curl-7579-f96b2a" Oct 30 03:39:16.394204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.394179 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnc2\" (UniqueName: \"kubernetes.io/projected/b14f496e-647e-4739-88c8-29f41f17b3ac-kube-api-access-qgnc2\") pod \"test-curl-7579-f96b2a\" (UID: \"b14f496e-647e-4739-88c8-29f41f17b3ac\") " pod="opendatahub/test-curl-7579-f96b2a" Oct 30 03:39:16.418958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.418926 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:16.543452 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.543416 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-f96b2a" Oct 30 03:39:16.854229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.854199 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-f96b2a"] Oct 30 03:39:16.855493 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:39:16.855468 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb14f496e_647e_4739_88c8_29f41f17b3ac.slice/crio-238a2bed96afcd6d191f8ff5ee60943ab4d869df5231081f58b25d4fa66610e0 WatchSource:0}: Error finding container 238a2bed96afcd6d191f8ff5ee60943ab4d869df5231081f58b25d4fa66610e0: Status 404 returned error can't find the container with id 238a2bed96afcd6d191f8ff5ee60943ab4d869df5231081f58b25d4fa66610e0 Oct 30 03:39:16.994648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:16.994607 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:17.138258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.138225 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-f96b2a" event={"ID":"b14f496e-647e-4739-88c8-29f41f17b3ac","Type":"ContainerStarted","Data":"faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b"} Oct 30 03:39:17.138651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.138268 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-f96b2a" event={"ID":"b14f496e-647e-4739-88c8-29f41f17b3ac","Type":"ContainerStarted","Data":"238a2bed96afcd6d191f8ff5ee60943ab4d869df5231081f58b25d4fa66610e0"} Oct 30 03:39:17.138651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.138284 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-18c4cb" podUID="d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd" containerName="test-curl-7579-18c4cb" containerID="cri-o://6304ed49c1748375cbe1f86d09d629426033f79cbaa0a435737593ae8a24fbe5" gracePeriod=2 Oct 30 03:39:17.178523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.175818 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-f96b2a" podStartSLOduration=1.175800386 podStartE2EDuration="1.175800386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:17.171426394 +0000 UTC m=+2181.173779592" watchObservedRunningTime="2026-10-30 03:39:17.175800386 +0000 UTC m=+2181.178153599" Oct 30 03:39:17.301780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.301738 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-a28246" Oct 30 03:39:17.387999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.387929 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae98857-42d4-4164-b468-309d4a6accfe-kube-api-access-29w5r\" (UniqueName: \"kubernetes.io/projected/2ae98857-42d4-4164-b468-309d4a6accfe-kube-api-access-29w5r\") pod \"2ae98857-42d4-4164-b468-309d4a6accfe\" (UID: \"2ae98857-42d4-4164-b468-309d4a6accfe\") " Oct 30 03:39:17.390161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.390131 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae98857-42d4-4164-b468-309d4a6accfe-kube-api-access-29w5r" pod "2ae98857-42d4-4164-b468-309d4a6accfe" (UID: "2ae98857-42d4-4164-b468-309d4a6accfe"). InnerVolumeSpecName "kube-api-access-29w5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:17.489315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.489287 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29w5r\" (UniqueName: \"kubernetes.io/projected/2ae98857-42d4-4164-b468-309d4a6accfe-kube-api-access-29w5r\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:39:17.998041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:17.998008 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:18.010688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.010661 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-f96b2a"] Oct 30 03:39:18.014911 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.014889 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-f96b2a"] Oct 30 03:39:18.109007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.108968 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae98857-42d4-4164-b468-309d4a6accfe" path="/var/lib/kubelet/pods/2ae98857-42d4-4164-b468-309d4a6accfe/volumes" Oct 30 03:39:18.157015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.156976 4442 generic.go:356] "Generic (PLEG): container finished" podID="2ae98857-42d4-4164-b468-309d4a6accfe" containerID="73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40" exitCode=137 Oct 30 03:39:18.158576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.158511 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-a28246" Oct 30 03:39:18.159030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.159006 4442 scope.go:122] "RemoveContainer" containerID="73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40" Oct 30 03:39:18.176454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.176427 4442 scope.go:122] "RemoveContainer" containerID="73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40" Oct 30 03:39:18.176877 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:39:18.176851 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40\": container with ID starting with 73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40 not found: ID does not exist" containerID="73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40" Oct 30 03:39:18.176991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.176883 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40"} err="failed to get container status \"73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40\": rpc error: code = NotFound desc = could not find container \"73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40\": container with ID starting with 73e11707e9d18ae2f82341c1662a203c4579aee975ba34ae3b9923f879db1b40 not found: ID does not exist" Oct 30 03:39:18.179477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.179456 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-fc70fd"] Oct 30 03:39:18.180056 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.180041 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b14f496e-647e-4739-88c8-29f41f17b3ac" containerName="test-curl-7579-f96b2a" Oct 30 03:39:18.180102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.180058 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14f496e-647e-4739-88c8-29f41f17b3ac" containerName="test-curl-7579-f96b2a" Oct 30 03:39:18.180146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.180135 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14f496e-647e-4739-88c8-29f41f17b3ac" containerName="test-curl-7579-f96b2a" Oct 30 03:39:18.184937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.184920 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-fc70fd" Oct 30 03:39:18.196819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.196797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ngbv\" (UniqueName: \"kubernetes.io/projected/2636d251-10aa-4f35-928d-4d5d82c3a285-kube-api-access-7ngbv\") pod \"test-curl-7579-fc70fd\" (UID: \"2636d251-10aa-4f35-928d-4d5d82c3a285\") " pod="opendatahub/test-curl-7579-fc70fd" Oct 30 03:39:18.199688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.199665 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-fc70fd"] Oct 30 03:39:18.297934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.297904 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ngbv\" (UniqueName: \"kubernetes.io/projected/2636d251-10aa-4f35-928d-4d5d82c3a285-kube-api-access-7ngbv\") pod \"test-curl-7579-fc70fd\" (UID: \"2636d251-10aa-4f35-928d-4d5d82c3a285\") " pod="opendatahub/test-curl-7579-fc70fd" Oct 30 03:39:18.315943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.315917 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ngbv\" (UniqueName: \"kubernetes.io/projected/2636d251-10aa-4f35-928d-4d5d82c3a285-kube-api-access-7ngbv\") pod \"test-curl-7579-fc70fd\" (UID: \"2636d251-10aa-4f35-928d-4d5d82c3a285\") " pod="opendatahub/test-curl-7579-fc70fd" Oct 30 03:39:18.498012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.497979 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-fc70fd" Oct 30 03:39:18.798805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:18.798781 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-fc70fd"] Oct 30 03:39:18.799657 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:39:18.799633 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2636d251_10aa_4f35_928d_4d5d82c3a285.slice/crio-2e306a02893db5a7a1acea15d85fdbcb8fe1fbc9e6c75203585b0465dcf11568 WatchSource:0}: Error finding container 2e306a02893db5a7a1acea15d85fdbcb8fe1fbc9e6c75203585b0465dcf11568: Status 404 returned error can't find the container with id 2e306a02893db5a7a1acea15d85fdbcb8fe1fbc9e6c75203585b0465dcf11568 Oct 30 03:39:19.176494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.176459 4442 generic.go:356] "Generic (PLEG): container finished" podID="d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd" containerID="6304ed49c1748375cbe1f86d09d629426033f79cbaa0a435737593ae8a24fbe5" exitCode=137 Oct 30 03:39:19.179205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.179169 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-fc70fd" event={"ID":"2636d251-10aa-4f35-928d-4d5d82c3a285","Type":"ContainerStarted","Data":"577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05"} Oct 30 03:39:19.179346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.179219 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-fc70fd" event={"ID":"2636d251-10aa-4f35-928d-4d5d82c3a285","Type":"ContainerStarted","Data":"2e306a02893db5a7a1acea15d85fdbcb8fe1fbc9e6c75203585b0465dcf11568"} Oct 30 03:39:19.179614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.179584 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-f96b2a" podUID="b14f496e-647e-4739-88c8-29f41f17b3ac" containerName="test-curl-7579-f96b2a" containerID="cri-o://faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b" gracePeriod=2 Oct 30 03:39:19.215948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.215892 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7579-fc70fd" podStartSLOduration=1.215878663 podStartE2EDuration="1.215878663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:19.214197856 +0000 UTC m=+2183.216551053" watchObservedRunningTime="2026-10-30 03:39:19.215878663 +0000 UTC m=+2183.218231859" Oct 30 03:39:19.311326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.311295 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-18c4cb" Oct 30 03:39:19.408774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.408670 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd-kube-api-access-7hj6h\" (UniqueName: \"kubernetes.io/projected/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd-kube-api-access-7hj6h\") pod \"d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd\" (UID: \"d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd\") " Oct 30 03:39:19.410905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.410878 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd-kube-api-access-7hj6h" pod "d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd" (UID: "d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd"). InnerVolumeSpecName "kube-api-access-7hj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:19.496965 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.496930 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:19.510135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:19.510092 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hj6h\" (UniqueName: \"kubernetes.io/projected/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd-kube-api-access-7hj6h\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:39:20.039632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:20.039597 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-fc70fd"] Oct 30 03:39:20.045333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:20.045298 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-fc70fd"] Oct 30 03:39:20.107116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:20.107089 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd" path="/var/lib/kubelet/pods/d7bb834a-50ff-4ebb-aeb5-a2d76f1039bd/volumes" Oct 30 03:39:20.186301 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:20.186277 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-18c4cb" Oct 30 03:39:20.186736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:20.186322 4442 scope.go:122] "RemoveContainer" containerID="6304ed49c1748375cbe1f86d09d629426033f79cbaa0a435737593ae8a24fbe5" Oct 30 03:39:20.623559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:20.623521 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:21.013928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:21.013901 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:21.193233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:21.193202 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7579-fc70fd" podUID="2636d251-10aa-4f35-928d-4d5d82c3a285" containerName="test-curl-7579-fc70fd" containerID="cri-o://577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05" gracePeriod=2 Oct 30 03:39:21.378089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:21.378066 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-f96b2a" Oct 30 03:39:21.430807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:21.430778 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14f496e-647e-4739-88c8-29f41f17b3ac-kube-api-access-qgnc2\" (UniqueName: \"kubernetes.io/projected/b14f496e-647e-4739-88c8-29f41f17b3ac-kube-api-access-qgnc2\") pod \"b14f496e-647e-4739-88c8-29f41f17b3ac\" (UID: \"b14f496e-647e-4739-88c8-29f41f17b3ac\") " Oct 30 03:39:21.432869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:21.432836 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14f496e-647e-4739-88c8-29f41f17b3ac-kube-api-access-qgnc2" pod "b14f496e-647e-4739-88c8-29f41f17b3ac" (UID: "b14f496e-647e-4739-88c8-29f41f17b3ac"). InnerVolumeSpecName "kube-api-access-qgnc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:21.532370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:21.532331 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnc2\" (UniqueName: \"kubernetes.io/projected/b14f496e-647e-4739-88c8-29f41f17b3ac-kube-api-access-qgnc2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:39:21.640471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:21.640395 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:22.107104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.107069 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14f496e-647e-4739-88c8-29f41f17b3ac" path="/var/lib/kubelet/pods/b14f496e-647e-4739-88c8-29f41f17b3ac/volumes" Oct 30 03:39:22.199867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.199827 4442 generic.go:356] "Generic (PLEG): container finished" podID="b14f496e-647e-4739-88c8-29f41f17b3ac" containerID="faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b" exitCode=137 Oct 30 03:39:22.200304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.199881 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-f96b2a" Oct 30 03:39:22.200304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.199894 4442 scope.go:122] "RemoveContainer" containerID="faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b" Oct 30 03:39:22.209007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.208982 4442 scope.go:122] "RemoveContainer" containerID="faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b" Oct 30 03:39:22.209308 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:39:22.209287 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b\": container with ID starting with faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b not found: ID does not exist" containerID="faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b" Oct 30 03:39:22.209385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.209320 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b"} err="failed to get container status \"faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b\": rpc error: code = NotFound desc = could not find container \"faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b\": container with ID starting with faef83cb796072a478e40e61817b38457fcc056f95e30a5d8482e078c183958b not found: ID does not exist" Oct 30 03:39:22.222683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.222660 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:22.402866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.402767 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:22.699106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:22.698996 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:23.404834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:23.404811 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-fc70fd" Oct 30 03:39:23.460277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:23.460205 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2636d251-10aa-4f35-928d-4d5d82c3a285-kube-api-access-7ngbv\" (UniqueName: \"kubernetes.io/projected/2636d251-10aa-4f35-928d-4d5d82c3a285-kube-api-access-7ngbv\") pod \"2636d251-10aa-4f35-928d-4d5d82c3a285\" (UID: \"2636d251-10aa-4f35-928d-4d5d82c3a285\") " Oct 30 03:39:23.462475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:23.462452 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2636d251-10aa-4f35-928d-4d5d82c3a285-kube-api-access-7ngbv" pod "2636d251-10aa-4f35-928d-4d5d82c3a285" (UID: "2636d251-10aa-4f35-928d-4d5d82c3a285"). InnerVolumeSpecName "kube-api-access-7ngbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:39:23.561290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:23.561259 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ngbv\" (UniqueName: \"kubernetes.io/projected/2636d251-10aa-4f35-928d-4d5d82c3a285-kube-api-access-7ngbv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:39:24.101371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:24.101320 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:24.107253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:24.107227 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2636d251-10aa-4f35-928d-4d5d82c3a285" path="/var/lib/kubelet/pods/2636d251-10aa-4f35-928d-4d5d82c3a285/volumes" Oct 30 03:39:24.213991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:24.213959 4442 generic.go:356] "Generic (PLEG): container finished" podID="2636d251-10aa-4f35-928d-4d5d82c3a285" containerID="577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05" exitCode=137 Oct 30 03:39:24.214192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:24.214017 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-fc70fd" Oct 30 03:39:24.214192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:24.214059 4442 scope.go:122] "RemoveContainer" containerID="577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05" Oct 30 03:39:24.223468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:24.223268 4442 scope.go:122] "RemoveContainer" containerID="577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05" Oct 30 03:39:24.223562 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:39:24.223545 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05\": container with ID starting with 577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05 not found: ID does not exist" containerID="577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05" Oct 30 03:39:24.223606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:24.223573 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05"} err="failed to get container status \"577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05\": rpc error: code = NotFound desc = could not find container \"577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05\": container with ID starting with 577feaf264b7c861f7200697ad96cf76b2c916056148a212a9e8bad988152a05 not found: ID does not exist" Oct 30 03:39:24.860904 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:39:24.860865 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:39:25.547423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:25.547388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:25.654209 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:25.654180 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:25.764491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:25.764456 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:27.001566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:27.001531 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:27.245212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:27.245175 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:27.342594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:27.342515 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:28.341010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:28.340976 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:28.961296 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:28.961260 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:29.062404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:29.062367 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:30.592109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:30.592075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:32.082270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:32.082232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:32.421169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:32.421088 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:33.612431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:33.612385 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:35.083853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:35.083826 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:36.010420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:36.010383 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:36.110801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:36.110747 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:36.480927 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:36.480845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:36.632900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:36.632868 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:36.737004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:36.736446 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:37.095916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:37.095878 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:37.188225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:37.188194 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:37.379053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:37.378965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:37.724977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:37.724896 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:38.134449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.134420 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks"] Oct 30 03:39:38.135204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.135183 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2636d251-10aa-4f35-928d-4d5d82c3a285" containerName="test-curl-7579-fc70fd" Oct 30 03:39:38.135316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.135206 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2636d251-10aa-4f35-928d-4d5d82c3a285" containerName="test-curl-7579-fc70fd" Oct 30 03:39:38.135380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.135368 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2636d251-10aa-4f35-928d-4d5d82c3a285" containerName="test-curl-7579-fc70fd" Oct 30 03:39:38.140936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.140916 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.156908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.156880 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks"] Oct 30 03:39:38.157147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.157127 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:38.166824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.166800 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-22aeb4-openshift-default-dockercfg-x786j\"" Oct 30 03:39:38.184676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.184645 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-certs\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.184807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.184709 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q477d\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-kube-api-access-q477d\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.184905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.184885 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c990cc-c7df-4b6b-a812-010138774fc8-istiod-ca-cert\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.184961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.184931 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-data\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.185064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.184981 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-token\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.185150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.185127 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-socket\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.185324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.185300 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-podinfo\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.185459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.185341 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-credential-socket\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.185533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.185491 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-envoy\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.273982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.273950 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:38.287035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.286995 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-data\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.287674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.287651 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-token\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.287948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.287853 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-socket\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.288123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.288103 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-podinfo\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.288238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.288223 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-credential-socket\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.288393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.288379 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-envoy\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.288515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.288501 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-certs\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.288636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.288622 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q477d\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-kube-api-access-q477d\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.288870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.288853 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c990cc-c7df-4b6b-a812-010138774fc8-istiod-ca-cert\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.289720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.289636 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c990cc-c7df-4b6b-a812-010138774fc8-istiod-ca-cert\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.298095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.287600 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-data\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.298247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.295661 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-socket\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.298350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.295900 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-credential-socket\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.298442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.297024 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-certs\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.319071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.318435 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-envoy\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.326272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.324921 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q477d\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-kube-api-access-q477d\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.334740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.330075 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-token\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.337555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.337525 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-podinfo\") pod \"e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.365289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.365262 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:38.456394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.456310 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:38.772788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.772738 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks"] Oct 30 03:39:38.774604 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:39:38.774577 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c990cc_c7df_4b6b_a812_010138774fc8.slice/crio-0f653e5a10d5cd0922fcbd10e548edcb582121a2960d52992fcca92384785905 WatchSource:0}: Error finding container 0f653e5a10d5cd0922fcbd10e548edcb582121a2960d52992fcca92384785905: Status 404 returned error can't find the container with id 0f653e5a10d5cd0922fcbd10e548edcb582121a2960d52992fcca92384785905 Oct 30 03:39:38.777113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.777083 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:39:38.777174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.777153 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:39:38.777211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:38.777184 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:39:39.351032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:39.350989 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" event={"ID":"f9c990cc-c7df-4b6b-a812-010138774fc8","Type":"ContainerStarted","Data":"b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3"} Oct 30 03:39:39.351454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:39.351055 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" event={"ID":"f9c990cc-c7df-4b6b-a812-010138774fc8","Type":"ContainerStarted","Data":"0f653e5a10d5cd0922fcbd10e548edcb582121a2960d52992fcca92384785905"} Oct 30 03:39:39.457316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:39.457286 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:39.463274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:39.463249 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:39.507278 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:39.507230 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" podStartSLOduration=1.5072158930000001 podStartE2EDuration="1.507215893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:39:39.41031045 +0000 UTC m=+2203.412663647" watchObservedRunningTime="2026-10-30 03:39:39.507215893 +0000 UTC m=+2203.509569089" Oct 30 03:39:39.755857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:39.755810 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:39.874436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:39.874391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:40.035957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:40.035880 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:40.357230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:40.357156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:40.358174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:40.358156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:39:40.863926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:40.863884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:40.963275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:40.963242 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:41.424014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:41.423981 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:42.455516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:42.455490 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:42.554717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:42.554685 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:43.180830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:43.180799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:44.692873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:44.692843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:46.195515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:46.195480 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:47.644523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:47.644489 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:47.748384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:47.748356 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:47.848265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:47.848229 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:48.666025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:48.665981 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:49.108220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:49.108190 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:49.681478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:49.681443 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:50.498276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:50.498247 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:50.611308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:50.611280 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:50.877451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:50.877416 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:39:52.181731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:52.181698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:52.387972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:52.387939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:53.694898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:53.694867 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:54.860747 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:39:54.860709 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:39:55.179413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:55.179333 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:56.691067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:56.691031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:58.087063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:58.087020 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:39:59.550563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:39:59.550533 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:00.567247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:00.567200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:00.738873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:00.738840 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:00.741387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:00.741355 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks"] Oct 30 03:40:00.741640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:00.741614 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" podUID="f9c990cc-c7df-4b6b-a812-010138774fc8" containerName="istio-proxy" containerID="cri-o://b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3" gracePeriod=30 Oct 30 03:40:01.039514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:01.039474 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:01.149790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:01.149732 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:01.241241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:01.241207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:02.423002 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:02.422973 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:02.500344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:02.500313 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:02.611428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:02.610483 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:03.916244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:03.916204 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:05.344398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:05.344367 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:06.006230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.006163 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:40:06.038086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038046 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-credential-socket\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038099 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c990cc-c7df-4b6b-a812-010138774fc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c990cc-c7df-4b6b-a812-010138774fc8-istiod-ca-cert\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038152 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-token\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-token\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038260 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-envoy\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038297 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-podinfo\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038332 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-credential-socket" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:06.038508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038367 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-data\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038463 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-socket\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038493 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c990cc-c7df-4b6b-a812-010138774fc8-istiod-ca-cert" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:40:06.038713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038503 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-certs\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038595 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-data" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:06.038713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038653 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-kube-api-access-q477d\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-kube-api-access-q477d\") pod \"f9c990cc-c7df-4b6b-a812-010138774fc8\" (UID: \"f9c990cc-c7df-4b6b-a812-010138774fc8\") " Oct 30 03:40:06.038898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038722 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-certs" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:06.038898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.038737 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-socket" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:06.039141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.039113 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.039270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.039144 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.039270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.039190 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.039391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.039258 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.039391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.039306 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9c990cc-c7df-4b6b-a812-010138774fc8-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.040537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.040514 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-envoy" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:06.040633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.040572 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-token" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:06.040711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.040646 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-podinfo" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:40:06.041045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.041017 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-kube-api-access-q477d" pod "f9c990cc-c7df-4b6b-a812-010138774fc8" (UID: "f9c990cc-c7df-4b6b-a812-010138774fc8"). InnerVolumeSpecName "kube-api-access-q477d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:06.140349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.140317 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.140349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.140340 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.140349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.140351 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q477d\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-kube-api-access-q477d\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.140574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.140362 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c990cc-c7df-4b6b-a812-010138774fc8-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:06.258565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.258490 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:06.508955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.508868 4442 generic.go:356] "Generic (PLEG): container finished" podID="f9c990cc-c7df-4b6b-a812-010138774fc8" containerID="b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3" exitCode=0 Oct 30 03:40:06.508955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.508915 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" Oct 30 03:40:06.509493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.508957 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" event={"ID":"f9c990cc-c7df-4b6b-a812-010138774fc8","Type":"ContainerDied","Data":"b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3"} Oct 30 03:40:06.509493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.509016 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks" event={"ID":"f9c990cc-c7df-4b6b-a812-010138774fc8","Type":"ContainerDied","Data":"0f653e5a10d5cd0922fcbd10e548edcb582121a2960d52992fcca92384785905"} Oct 30 03:40:06.509493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.509042 4442 scope.go:122] "RemoveContainer" containerID="b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3" Oct 30 03:40:06.519234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.519214 4442 scope.go:122] "RemoveContainer" containerID="b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3" Oct 30 03:40:06.519536 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:40:06.519514 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3\": container with ID starting with b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3 not found: ID does not exist" containerID="b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3" Oct 30 03:40:06.519589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.519545 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3"} err="failed to get container status \"b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3\": rpc error: code = NotFound desc = could not find container \"b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3\": container with ID starting with b078f3a38b4639cb7fdc5f9eaf75bb74807454629b828598dad6d4046c9e2eb3 not found: ID does not exist" Oct 30 03:40:06.555359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.555334 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks"] Oct 30 03:40:06.565953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.565922 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-22aeb4-openshift-default-698fcbb6bf-7ktks"] Oct 30 03:40:06.744979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:06.744945 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:08.107723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:08.107689 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c990cc-c7df-4b6b-a812-010138774fc8" path="/var/lib/kubelet/pods/f9c990cc-c7df-4b6b-a812-010138774fc8/volumes" Oct 30 03:40:08.242770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:08.242730 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:09.774211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:09.774178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:11.197424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:11.197381 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:12.695445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:12.695404 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:14.211537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:14.211504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:15.597420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:15.597389 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:16.545347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:16.545294 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:17.203739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:17.203699 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:18.023185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:18.023135 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:18.407706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:18.407630 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:18.683859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:18.683785 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:18.784524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:18.784485 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:18.936536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:18.936459 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:20.288260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:20.288223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:22.027438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:22.027397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:22.147270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:22.147234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:22.254034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:22.254002 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:23.079411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:23.079370 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:23.691862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:23.691831 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:24.615269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:24.615225 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:24.864410 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:40:24.864377 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:40:24.897961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:24.897882 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:24.997793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:24.997744 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:25.032129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.032100 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j"] Oct 30 03:40:25.032886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.032865 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f9c990cc-c7df-4b6b-a812-010138774fc8" containerName="istio-proxy" Oct 30 03:40:25.032886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.032884 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c990cc-c7df-4b6b-a812-010138774fc8" containerName="istio-proxy" Oct 30 03:40:25.033093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.033059 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c990cc-c7df-4b6b-a812-010138774fc8" containerName="istio-proxy" Oct 30 03:40:25.036712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.036691 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.055252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.055228 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j"] Oct 30 03:40:25.062728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.062687 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-35f43f\"" Oct 30 03:40:25.062942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.062855 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-35f43f-dockercfg-4vtgh\"" Oct 30 03:40:25.091257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.091233 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:25.167467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.167381 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhqwm\" (UniqueName: \"kubernetes.io/projected/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-kube-api-access-bhqwm\") pod \"payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.167814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.167532 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.171727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.171687 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:25.268638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.268601 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhqwm\" (UniqueName: \"kubernetes.io/projected/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-kube-api-access-bhqwm\") pod \"payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.268856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.268700 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.269503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.269478 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.271926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.271892 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:25.302107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.302009 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhqwm\" (UniqueName: \"kubernetes.io/projected/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-kube-api-access-bhqwm\") pod \"payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.351929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.351886 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:25.650356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:25.650326 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j"] Oct 30 03:40:25.651864 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:40:25.651829 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdb7dd7_0fa4_45e8_995e_2ec2238049c5.slice/crio-ffdb67ff7753e7647060d481f0b075d97288091b12ff660d89a9b6c6b17e5196 WatchSource:0}: Error finding container ffdb67ff7753e7647060d481f0b075d97288091b12ff660d89a9b6c6b17e5196: Status 404 returned error can't find the container with id ffdb67ff7753e7647060d481f0b075d97288091b12ff660d89a9b6c6b17e5196 Oct 30 03:40:26.518780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:26.518728 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:26.647423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:26.647392 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" event={"ID":"acdb7dd7-0fa4-45e8-995e-2ec2238049c5","Type":"ContainerStarted","Data":"045466d43cdeb20d275fa8a1b7c4c2696422af83c2b94fbcff5c3d7cd2187dfa"} Oct 30 03:40:26.647622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:26.647447 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" event={"ID":"acdb7dd7-0fa4-45e8-995e-2ec2238049c5","Type":"ContainerStarted","Data":"ffdb67ff7753e7647060d481f0b075d97288091b12ff660d89a9b6c6b17e5196"} Oct 30 03:40:26.647622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:26.647491 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:26.684556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:26.684502 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" podStartSLOduration=1.6844898069999998 podStartE2EDuration="1.684489807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:26.682429747 +0000 UTC m=+2250.684782945" watchObservedRunningTime="2026-10-30 03:40:26.684489807 +0000 UTC m=+2250.686843034" Oct 30 03:40:28.110991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:28.110957 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:28.216546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:28.216510 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:28.319185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:28.319131 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:29.846586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:29.846548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:29.943900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:29.943862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:30.029425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.029393 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59"] Oct 30 03:40:30.033683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.033660 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.053593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.053566 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59"] Oct 30 03:40:30.090698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.090662 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-e10c0f\"" Oct 30 03:40:30.131952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.131873 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-e10c0f-dockercfg-mqxdc\"" Oct 30 03:40:30.147299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.147267 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhfj\" (UniqueName: \"kubernetes.io/projected/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-kube-api-access-pmhfj\") pod \"payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.147889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.147806 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.250746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.250712 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.250964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.250857 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhfj\" (UniqueName: \"kubernetes.io/projected/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-kube-api-access-pmhfj\") pod \"payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.251581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.251552 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.268903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.268878 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhfj\" (UniqueName: \"kubernetes.io/projected/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-kube-api-access-pmhfj\") pod \"payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.347299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.347269 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:30.643281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.643255 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59"] Oct 30 03:40:30.644072 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:40:30.644047 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b33eb0_d5c0_4160_8a3f_3c53d63f5a14.slice/crio-816dabf3fc741cd4d5f819089b722f8941598b099bc867cf30c76e3ad5c3e2a4 WatchSource:0}: Error finding container 816dabf3fc741cd4d5f819089b722f8941598b099bc867cf30c76e3ad5c3e2a4: Status 404 returned error can't find the container with id 816dabf3fc741cd4d5f819089b722f8941598b099bc867cf30c76e3ad5c3e2a4 Oct 30 03:40:30.672730 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:30.672697 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" event={"ID":"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14","Type":"ContainerStarted","Data":"816dabf3fc741cd4d5f819089b722f8941598b099bc867cf30c76e3ad5c3e2a4"} Oct 30 03:40:31.329131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:31.329095 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:31.679765 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:31.679648 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" event={"ID":"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14","Type":"ContainerStarted","Data":"12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571"} Oct 30 03:40:31.679765 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:31.679727 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:31.716077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:31.716033 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" podStartSLOduration=1.71601464 podStartE2EDuration="1.71601464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:31.713891136 +0000 UTC m=+2255.716244335" watchObservedRunningTime="2026-10-30 03:40:31.71601464 +0000 UTC m=+2255.718367831" Oct 30 03:40:31.776729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:31.776692 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:32.744294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:32.744265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:33.142834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:33.142803 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:33.205738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:33.205705 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2"] Oct 30 03:40:33.235483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:33.235458 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:33.681030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:33.680974 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:33.931596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:33.931507 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:34.399101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:34.399070 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:35.054834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:35.054805 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:35.903441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:35.903401 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:36.141899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:36.141868 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:36.782232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:36.782199 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:37.178026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.177948 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j"] Oct 30 03:40:37.183127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.183096 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.200676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.200654 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-dbed6d-openshift-default-dockercfg-2cjtq\"" Oct 30 03:40:37.218403 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.218359 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j"] Oct 30 03:40:37.229141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.229101 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:37.252641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.252616 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-data\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.252784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.252647 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-certs\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.252784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.252675 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-credential-socket\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.252859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.252794 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-podinfo\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.252894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.252866 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643fe397-ece5-48d2-96cf-a088d3856eb3-istiod-ca-cert\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.252936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.252900 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-token\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.253042 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.253028 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-socket\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.253078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.253054 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-envoy\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.253114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.253086 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cp67\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-kube-api-access-9cp67\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.354474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354432 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cp67\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-kube-api-access-9cp67\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.354685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354548 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-data\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.354685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354581 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-certs\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.354685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354612 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-credential-socket\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.354685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354642 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-podinfo\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.354685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354678 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643fe397-ece5-48d2-96cf-a088d3856eb3-istiod-ca-cert\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.355003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354723 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-token\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.355003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354915 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-socket\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.355003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.354951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-envoy\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.355415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.355387 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-credential-socket\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.356838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.355975 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-data\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.357012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.356119 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643fe397-ece5-48d2-96cf-a088d3856eb3-istiod-ca-cert\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.357157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.356995 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-socket\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.366922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.360446 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-envoy\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.366922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.360993 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-certs\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.375471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.374902 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-podinfo\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.378972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.378937 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-token\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.383770 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.381124 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cp67\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-kube-api-access-9cp67\") pod \"e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.479212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.478850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:37.498889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.498859 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:37.584127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.583087 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:37.694787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.693845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:37.710496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.709360 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 03:40:37.832076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.832052 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j"] Oct 30 03:40:37.832807 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:40:37.832782 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643fe397_ece5_48d2_96cf_a088d3856eb3.slice/crio-cddea890cb3d594fef0a0685e8070744742faccb3d199c72271b6ddbe8b8ee0e WatchSource:0}: Error finding container cddea890cb3d594fef0a0685e8070744742faccb3d199c72271b6ddbe8b8ee0e: Status 404 returned error can't find the container with id cddea890cb3d594fef0a0685e8070744742faccb3d199c72271b6ddbe8b8ee0e Oct 30 03:40:37.835365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.835335 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:40:37.835429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.835402 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:40:37.835466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:37.835432 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:40:38.737732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:38.737695 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" event={"ID":"643fe397-ece5-48d2-96cf-a088d3856eb3","Type":"ContainerStarted","Data":"ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b"} Oct 30 03:40:38.738144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:38.737805 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" event={"ID":"643fe397-ece5-48d2-96cf-a088d3856eb3","Type":"ContainerStarted","Data":"cddea890cb3d594fef0a0685e8070744742faccb3d199c72271b6ddbe8b8ee0e"} Oct 30 03:40:38.777871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:38.777804 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" podStartSLOduration=1.7777865880000001 podStartE2EDuration="1.777786588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:38.776516395 +0000 UTC m=+2262.778869593" watchObservedRunningTime="2026-10-30 03:40:38.777786588 +0000 UTC m=+2262.780139786" Oct 30 03:40:39.167461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:39.167433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:39.276234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:39.276200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:39.378952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:39.378917 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:39.509829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:39.509732 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:39.574658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:39.574631 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:39.744022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:39.743988 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:39.744886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:39.744869 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:40:41.009160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:41.009128 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:41.110370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:41.110338 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:41.210219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:41.210191 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:42.605190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:42.605158 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:42.692669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:42.692641 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 03:40:42.800198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:42.800170 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:42.808563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:42.808530 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:40:42.942517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:42.942430 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:44.335691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:44.335656 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:44.716410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:44.716327 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:45.899046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:45.899016 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:46.167364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:46.167290 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:46.660933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:46.660901 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:46.819352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:46.819315 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:47.351528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:47.351498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:47.454543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:47.454511 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:48.241075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.241004 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" podUID="a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" containerName="main" containerID="cri-o://ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f" gracePeriod=45 Oct 30 03:40:48.519187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.519164 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:40:48.536322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536293 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tmp-dir\") pod \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " Oct 30 03:40:48.536444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536374 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-home\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-home\") pod \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " Oct 30 03:40:48.536444 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536417 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tls-certs\") pod \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " Oct 30 03:40:48.536529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536467 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tmp-dir" pod "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" (UID: "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:48.536570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536525 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-dshm\") pod \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " Oct 30 03:40:48.536614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536572 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-kube-api-access-rb5df\") pod \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " Oct 30 03:40:48.536651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536610 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-model-cache\") pod \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\" (UID: \"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e\") " Oct 30 03:40:48.536651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536639 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-home" pod "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" (UID: "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:48.536993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.536967 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-model-cache" pod "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" (UID: "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:48.537258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.537074 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:48.537258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.537106 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:48.537258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.537118 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:48.538866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.538828 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-kube-api-access-rb5df" pod "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" (UID: "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e"). InnerVolumeSpecName "kube-api-access-rb5df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:40:48.538969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.538906 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-dshm" pod "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" (UID: "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:40:48.538969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.538925 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tls-certs" pod "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" (UID: "a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:40:48.638566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.638531 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:48.638566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.638563 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:48.638566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.638574 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb5df\" (UniqueName: \"kubernetes.io/projected/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e-kube-api-access-rb5df\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:40:48.772889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.772857 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:48.811875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.811844 4442 generic.go:356] "Generic (PLEG): container finished" podID="a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" containerID="ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f" exitCode=0 Oct 30 03:40:48.812032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.811891 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" Oct 30 03:40:48.812032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.811931 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" event={"ID":"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e","Type":"ContainerDied","Data":"ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f"} Oct 30 03:40:48.812032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.811977 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2" event={"ID":"a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e","Type":"ContainerDied","Data":"32809a7c3457d24c381ff99fe4fcea68417be0fc4157910a75424367c8eb5932"} Oct 30 03:40:48.812032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.811999 4442 scope.go:122] "RemoveContainer" containerID="ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f" Oct 30 03:40:48.822639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.822623 4442 scope.go:122] "RemoveContainer" containerID="ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f" Oct 30 03:40:48.823008 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:40:48.822982 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f\": container with ID starting with ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f not found: ID does not exist" containerID="ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f" Oct 30 03:40:48.823096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.823018 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f"} err="failed to get container status \"ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f\": rpc error: code = NotFound desc = could not find container \"ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f\": container with ID starting with ccbff848877cdc31d2a1097754468dc36d8a4a33e1e9215298bd19a4a2fcd27f not found: ID does not exist" Oct 30 03:40:48.858020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.857997 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2"] Oct 30 03:40:48.858905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.858888 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:48.860472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:48.860453 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6ad168/auth-test-model-6ad168-kserve-dc644cfd9-n6pb2"] Oct 30 03:40:49.789132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:49.789097 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:50.108213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:50.108128 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" path="/var/lib/kubelet/pods/a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e/volumes" Oct 30 03:40:50.483130 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:50.483050 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:50.731616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:50.731588 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:50.858609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:50.858570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:50.959094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:50.959065 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:52.372550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.372511 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:52.472287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.472253 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:52.555362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.555330 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:52.557171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.557146 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s"] Oct 30 03:40:52.559025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.558651 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" containerName="main" Oct 30 03:40:52.559025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.558676 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" containerName="main" Oct 30 03:40:52.559025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.558967 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9efaf3a-5f36-4d00-bf14-a0a052e7ac7e" containerName="main" Oct 30 03:40:52.568264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.568239 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:52.578316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.578293 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s"] Oct 30 03:40:52.588780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.585596 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-398c25\"" Oct 30 03:40:52.588780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.585698 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-398c25-dockercfg-zc4nr\"" Oct 30 03:40:52.597768 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.597723 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:52.597935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.597916 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7stm\" (UniqueName: \"kubernetes.io/projected/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-kube-api-access-j7stm\") pod \"payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:52.736325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.720627 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7stm\" (UniqueName: \"kubernetes.io/projected/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-kube-api-access-j7stm\") pod \"payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:52.737128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.737085 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:52.738574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.738201 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:52.782748 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.782715 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7stm\" (UniqueName: \"kubernetes.io/projected/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-kube-api-access-j7stm\") pod \"payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:52.884555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:52.884526 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:53.188258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:53.188229 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s"] Oct 30 03:40:53.189302 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:40:53.189276 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1480f37f_0d88_4fcf_8a8f_0a18a11cfc29.slice/crio-3feb8feb07ae8ec3c09c2f9404def923af3c173c4a2920962cb234c7a2f0258d WatchSource:0}: Error finding container 3feb8feb07ae8ec3c09c2f9404def923af3c173c4a2920962cb234c7a2f0258d: Status 404 returned error can't find the container with id 3feb8feb07ae8ec3c09c2f9404def923af3c173c4a2920962cb234c7a2f0258d Oct 30 03:40:53.843640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:53.843605 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" event={"ID":"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29","Type":"ContainerStarted","Data":"9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7"} Oct 30 03:40:53.844101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:53.843661 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" event={"ID":"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29","Type":"ContainerStarted","Data":"3feb8feb07ae8ec3c09c2f9404def923af3c173c4a2920962cb234c7a2f0258d"} Oct 30 03:40:53.844101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:53.843781 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:40:53.886152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:53.886102 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" podStartSLOduration=1.886087581 podStartE2EDuration="1.886087581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:53.877715376 +0000 UTC m=+2277.880068574" watchObservedRunningTime="2026-10-30 03:40:53.886087581 +0000 UTC m=+2277.888440778" Oct 30 03:40:53.915438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:53.915410 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:54.860786 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:40:54.860723 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:40:55.303468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:55.303441 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:55.383988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:55.383962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:55.493737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:55.487157 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:56.910588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:56.910560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:58.481177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:58.481145 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:40:59.981653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:40:59.981619 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:01.474454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:01.474418 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:02.914858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:02.914830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:04.376240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:04.376205 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:04.853767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:04.853735 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 03:41:05.766181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:05.766136 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:05.866976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:05.866949 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:05.966733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:05.966688 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:07.275648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:07.275615 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:08.893384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:08.893353 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:10.280338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.280276 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj"] Oct 30 03:41:10.286131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.286108 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.303658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.303629 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj"] Oct 30 03:41:10.310552 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.310528 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-dbed6d-serving-cert\"" Oct 30 03:41:10.357152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.357125 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:10.405582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.405549 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307897d1-e14b-466c-ad08-97e765eae743-openshift-service-ca\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.405790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.405695 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mnk\" (UniqueName: \"kubernetes.io/projected/307897d1-e14b-466c-ad08-97e765eae743-kube-api-access-w4mnk\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.405790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.405732 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-maas-api-tls\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.405960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.405868 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-metrics-certs\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.456855 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.456827 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:10.507666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.507541 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307897d1-e14b-466c-ad08-97e765eae743-openshift-service-ca\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.512895 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.511419 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307897d1-e14b-466c-ad08-97e765eae743-openshift-service-ca\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.521692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.512929 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mnk\" (UniqueName: \"kubernetes.io/projected/307897d1-e14b-466c-ad08-97e765eae743-kube-api-access-w4mnk\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.521692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.517915 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-maas-api-tls\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.521692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.517989 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-metrics-certs\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.535154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.535068 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-maas-api-tls\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.536245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.536220 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-metrics-certs\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.536842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.536792 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mnk\" (UniqueName: \"kubernetes.io/projected/307897d1-e14b-466c-ad08-97e765eae743-kube-api-access-w4mnk\") pod \"maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.556384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.556354 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:10.599735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.599707 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:10.898525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.898502 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj"] Oct 30 03:41:10.900174 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:41:10.900146 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307897d1_e14b_466c_ad08_97e765eae743.slice/crio-c7cdc1f2cd5b556174b2637bd8390c50186adc533d66a76194abdff21a63f70f WatchSource:0}: Error finding container c7cdc1f2cd5b556174b2637bd8390c50186adc533d66a76194abdff21a63f70f: Status 404 returned error can't find the container with id c7cdc1f2cd5b556174b2637bd8390c50186adc533d66a76194abdff21a63f70f Oct 30 03:41:10.901925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:10.901903 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:41:11.014870 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:11.014816 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" event={"ID":"307897d1-e14b-466c-ad08-97e765eae743","Type":"ContainerStarted","Data":"c7cdc1f2cd5b556174b2637bd8390c50186adc533d66a76194abdff21a63f70f"} Oct 30 03:41:11.896604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:11.896570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:12.023670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:12.023639 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" event={"ID":"307897d1-e14b-466c-ad08-97e765eae743","Type":"ContainerStarted","Data":"42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7"} Oct 30 03:41:12.023887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:12.023793 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:12.062068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:12.062022 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" podStartSLOduration=1.6692272190000002 podStartE2EDuration="2.062008907s" totalImagesPullingTime="392.781688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:12.060262046 +0000 UTC m=+2296.062615244" watchObservedRunningTime="2026-10-30 03:41:12.062008907 +0000 UTC m=+2296.064362104" Oct 30 03:41:13.276230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:13.276196 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:14.700292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:14.700257 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:16.203221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:16.203191 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:17.843605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:17.843559 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:18.034979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:18.034949 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:41:19.278734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:19.278700 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:20.881636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:20.881588 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:20.965344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:20.965311 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:21.073821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:21.070638 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:22.621678 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:22.621620 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:23.365097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:23.365060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:24.204503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:24.204469 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:24.575879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:24.575843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:24.678554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:24.678523 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:24.860810 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:41:24.860701 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:41:24.865441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:24.865419 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:25.135455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:25.135352 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:25.290523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:25.290486 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:25.436806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:25.436696 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:25.916360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:25.916329 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:26.207426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:26.207354 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:26.351820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:26.343136 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:27.367253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:27.367219 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:27.468548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:27.468517 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:28.886683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:28.886645 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:29.383208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:29.383172 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:30.386144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:30.386106 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:30.730329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:30.730251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:31.209252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:31.209216 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:31.311116 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:31.311085 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:31.826197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:31.826166 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:33.199435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:33.199407 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:33.799853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:33.799810 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:41:34.827622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:34.827592 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:35.398973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:35.398931 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:36.016733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:36.016686 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:36.607459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:36.607423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:37.063176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.063144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:37.163300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.163267 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:37.257315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.257284 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj"] Oct 30 03:41:37.261807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.261789 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.293107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.293078 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-372480-dockercfg-wlr5h\"" Oct 30 03:41:37.294879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.294854 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-372480\"" Oct 30 03:41:37.306838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.306813 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj"] Oct 30 03:41:37.394884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.394798 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2"] Oct 30 03:41:37.453780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.428886 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7tr7\" (UniqueName: \"kubernetes.io/projected/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-kube-api-access-c7tr7\") pod \"payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.453780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.428963 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.473059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.463422 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2"] Oct 30 03:41:37.473059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.463554 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.530586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.530556 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7tr7\" (UniqueName: \"kubernetes.io/projected/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-kube-api-access-c7tr7\") pod \"payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.530776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.530612 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.532420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.532389 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.548404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.548377 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7tr7\" (UniqueName: \"kubernetes.io/projected/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-kube-api-access-c7tr7\") pod \"payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.575380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.575353 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:37.632458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.632420 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c7cd60-845b-4415-a560-1e913027eeb6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.632610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.632583 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwkqf\" (UniqueName: \"kubernetes.io/projected/60c7cd60-845b-4415-a560-1e913027eeb6-kube-api-access-pwkqf\") pod \"payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.734347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.734305 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c7cd60-845b-4415-a560-1e913027eeb6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.734562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.734388 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwkqf\" (UniqueName: \"kubernetes.io/projected/60c7cd60-845b-4415-a560-1e913027eeb6-kube-api-access-pwkqf\") pod \"payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.735161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.735129 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c7cd60-845b-4415-a560-1e913027eeb6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.752469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.752437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwkqf\" (UniqueName: \"kubernetes.io/projected/60c7cd60-845b-4415-a560-1e913027eeb6-kube-api-access-pwkqf\") pod \"payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.823990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.823958 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:37.921309 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:41:37.921251 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf993ec35_1ad1_4ab0_b5a4_eebc93dd66ca.slice/crio-c5ec2ec057752f5e8cabba913200f8eba04d970423c9f0353698380fef6600ef WatchSource:0}: Error finding container c5ec2ec057752f5e8cabba913200f8eba04d970423c9f0353698380fef6600ef: Status 404 returned error can't find the container with id c5ec2ec057752f5e8cabba913200f8eba04d970423c9f0353698380fef6600ef Oct 30 03:41:37.924216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:37.924129 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj"] Oct 30 03:41:38.153124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:38.153093 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2"] Oct 30 03:41:38.153462 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:41:38.153132 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c7cd60_845b_4415_a560_1e913027eeb6.slice/crio-34cf5ab63c147652aa56f29d101e9492f6f67c8d5ec1119e34396199d95ce291 WatchSource:0}: Error finding container 34cf5ab63c147652aa56f29d101e9492f6f67c8d5ec1119e34396199d95ce291: Status 404 returned error can't find the container with id 34cf5ab63c147652aa56f29d101e9492f6f67c8d5ec1119e34396199d95ce291 Oct 30 03:41:38.213069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:38.213040 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" event={"ID":"60c7cd60-845b-4415-a560-1e913027eeb6","Type":"ContainerStarted","Data":"34cf5ab63c147652aa56f29d101e9492f6f67c8d5ec1119e34396199d95ce291"} Oct 30 03:41:38.214715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:38.214689 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" event={"ID":"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca","Type":"ContainerStarted","Data":"598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4"} Oct 30 03:41:38.214858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:38.214738 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" event={"ID":"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca","Type":"ContainerStarted","Data":"c5ec2ec057752f5e8cabba913200f8eba04d970423c9f0353698380fef6600ef"} Oct 30 03:41:38.214926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:38.214912 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:38.252011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:38.251957 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" podStartSLOduration=1.251939955 podStartE2EDuration="1.251939955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:38.250604791 +0000 UTC m=+2322.252957989" watchObservedRunningTime="2026-10-30 03:41:38.251939955 +0000 UTC m=+2322.254293154" Oct 30 03:41:38.278236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:38.278214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:39.222686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:39.222634 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" event={"ID":"60c7cd60-845b-4415-a560-1e913027eeb6","Type":"ContainerStarted","Data":"c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723"} Oct 30 03:41:39.223175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:39.222859 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:39.259221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:39.259169 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" podStartSLOduration=2.259154294 podStartE2EDuration="2.259154294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:39.257220996 +0000 UTC m=+2323.259574189" watchObservedRunningTime="2026-10-30 03:41:39.259154294 +0000 UTC m=+2323.261507490" Oct 30 03:41:39.909004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:39.908969 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:41.486616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:41.486561 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:42.911038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:42.911001 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:43.821412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:43.821376 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:44.657565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:44.657533 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:45.136039 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.135971 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:45.238214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.238126 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:45.263820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.263789 4442 generic.go:356] "Generic (PLEG): container finished" podID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerID="ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b" exitCode=0 Oct 30 03:41:45.263973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.263840 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" event={"ID":"643fe397-ece5-48d2-96cf-a088d3856eb3","Type":"ContainerDied","Data":"ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b"} Oct 30 03:41:45.264216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.264201 4442 scope.go:122] "RemoveContainer" containerID="ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b" Oct 30 03:41:45.265897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.265871 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:41:45.265989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.265940 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:41:45.265989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.265986 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:41:45.365133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.365081 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:45.686018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.685982 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:45.843583 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.843553 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:45.947188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:45.947085 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:46.271540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:46.271506 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" event={"ID":"643fe397-ece5-48d2-96cf-a088d3856eb3","Type":"ContainerStarted","Data":"fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581"} Oct 30 03:41:46.402251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:46.402218 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:46.500179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:46.499961 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:41:46.500815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:46.500788 4442 patch_prober.go:29] interesting pod/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:41:46.501045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:46.501010 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 03:41:46.531904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:46.531817 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:46.634070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:46.634039 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:47.379044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:47.379010 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:47.501883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:47.501534 4442 patch_prober.go:29] interesting pod/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:41:47.501883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:47.501630 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 03:41:48.012773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.008542 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:48.151225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.151192 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:48.272859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.272777 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh"] Oct 30 03:41:48.275728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.275708 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.321585 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.321556 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh"] Oct 30 03:41:48.323474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.323453 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.323566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.323485 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.323603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.323593 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh8b9\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-kube-api-access-sh8b9\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.323738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.323723 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.329094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.329073 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93b821\"/\"agc-cred-8522e860b8fb\"" Oct 30 03:41:48.329219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.329197 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-93b821\"/\"payload-processing-external-model-e2e--c31d873d-dockercfg-h2b5q\"" Oct 30 03:41:48.329291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.329256 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93b821\"/\"kube-root-ca.crt\"" Oct 30 03:41:48.329291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.329263 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93b821\"/\"openshift-service-ca.crt\"" Oct 30 03:41:48.329291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.329286 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93b821\"/\"payload-processing-external-model-plugins-e2e-extmod-b-93b821\"" Oct 30 03:41:48.329416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.329306 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-93b821\"/\"routing-overlay\"" Oct 30 03:41:48.425031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.424994 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.425808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.425668 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.425808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.425717 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.426829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.426798 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh8b9\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-kube-api-access-sh8b9\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.431864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.427021 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.431864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.427483 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.431864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.428897 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.470788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.451932 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh8b9\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-kube-api-access-sh8b9\") pod \"payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.510777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.507147 4442 patch_prober.go:29] interesting pod/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:41:48.510777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.507224 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 03:41:48.619905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.619800 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:48.925107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.925003 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:48.947208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:48.947179 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh"] Oct 30 03:41:48.947466 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:41:48.947441 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5cea7f_a393_4170_a641_af2768a33cca.slice/crio-5229d4be16edcb1faba6dd2176174ef675f57663dc11c7ea778de741d8156cf5 WatchSource:0}: Error finding container 5229d4be16edcb1faba6dd2176174ef675f57663dc11c7ea778de741d8156cf5: Status 404 returned error can't find the container with id 5229d4be16edcb1faba6dd2176174ef675f57663dc11c7ea778de741d8156cf5 Oct 30 03:41:49.019682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.019651 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:49.226233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.226133 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 03:41:49.325229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.325191 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" event={"ID":"fc5cea7f-a393-4170-a641-af2768a33cca","Type":"ContainerStarted","Data":"3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39"} Oct 30 03:41:49.325410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.325264 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:41:49.325410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.325285 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" event={"ID":"fc5cea7f-a393-4170-a641-af2768a33cca","Type":"ContainerStarted","Data":"5229d4be16edcb1faba6dd2176174ef675f57663dc11c7ea778de741d8156cf5"} Oct 30 03:41:49.362195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.362147 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" podStartSLOduration=1.3621341980000001 podStartE2EDuration="1.362134198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:49.360184187 +0000 UTC m=+2333.362537399" watchObservedRunningTime="2026-10-30 03:41:49.362134198 +0000 UTC m=+2333.364487395" Oct 30 03:41:49.504050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.503166 4442 patch_prober.go:29] interesting pod/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:41:49.504050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.503257 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 03:41:49.812025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.811939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:49.953799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:49.953743 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:50.232058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:50.231978 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 03:41:50.507264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:50.507155 4442 patch_prober.go:29] interesting pod/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 03:41:50.507264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:50.507214 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 03:41:50.931543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:50.931466 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:51.314268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:51.314215 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:51.597097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:51.597018 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:41:51.597595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:51.597557 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:41:51.598456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:51.598437 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:41:52.804015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:52.803959 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:54.198266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:54.198237 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:54.860968 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:41:54.860933 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:41:55.627297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:55.627265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:57.090584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:57.090554 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:57.194872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:57.194843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:58.407876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:58.407847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:58.657526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:58.657493 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:58.880057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:58.880018 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:41:58.967881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:41:58.967843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:00.202305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:00.202273 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:00.336095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:00.336061 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:42:00.572000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:00.571968 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:42:01.730178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:01.730147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:02.904070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:02.904034 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:03.003510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:03.003476 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:03.567108 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:03.567080 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:04.575355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.575316 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:04.678801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.678763 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:04.706574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.706548 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2"] Oct 30 03:42:04.709595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.709572 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:04.730379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.730353 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2"] Oct 30 03:42:04.734408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.734388 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-855e14\"" Oct 30 03:42:04.877840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.875948 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pm4\" (UniqueName: \"kubernetes.io/projected/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-kube-api-access-x5pm4\") pod \"payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:04.877840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.876033 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:04.977043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.976999 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pm4\" (UniqueName: \"kubernetes.io/projected/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-kube-api-access-x5pm4\") pod \"payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:04.977400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.977368 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:04.978626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.978595 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:04.996064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:04.996035 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pm4\" (UniqueName: \"kubernetes.io/projected/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-kube-api-access-x5pm4\") pod \"payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:05.022289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:05.022260 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:05.187654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:05.187574 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:05.335825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:05.335794 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2"] Oct 30 03:42:05.336616 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:42:05.336596 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe013b5e_3ce2_41f1_860d_7ce69690c0a8.slice/crio-766996063c8984685fbe0e4d32aa91528d9c5dcf9808c82462c72143eaf76da5 WatchSource:0}: Error finding container 766996063c8984685fbe0e4d32aa91528d9c5dcf9808c82462c72143eaf76da5: Status 404 returned error can't find the container with id 766996063c8984685fbe0e4d32aa91528d9c5dcf9808c82462c72143eaf76da5 Oct 30 03:42:05.427422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:05.427385 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" event={"ID":"fe013b5e-3ce2-41f1-860d-7ce69690c0a8","Type":"ContainerStarted","Data":"1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a"} Oct 30 03:42:05.427539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:05.427452 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:05.427539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:05.427469 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" event={"ID":"fe013b5e-3ce2-41f1-860d-7ce69690c0a8","Type":"ContainerStarted","Data":"766996063c8984685fbe0e4d32aa91528d9c5dcf9808c82462c72143eaf76da5"} Oct 30 03:42:05.465920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:05.465821 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" podStartSLOduration=1.465803867 podStartE2EDuration="1.465803867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:05.462262173 +0000 UTC m=+2349.464615371" watchObservedRunningTime="2026-10-30 03:42:05.465803867 +0000 UTC m=+2349.468157081" Oct 30 03:42:06.618145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:06.618113 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:08.075489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.075457 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk"] Oct 30 03:42:08.078361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.078345 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.095167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.095139 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-372480-serving-cert\"" Oct 30 03:42:08.109235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.109208 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk"] Oct 30 03:42:08.141581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.141550 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:08.216923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.216899 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:08.218196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.218170 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-metrics-certs\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.218311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.218226 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7867e74-3d84-4780-88b8-a78222af9a95-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.218355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.218329 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-maas-api-tls\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.218392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.218353 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trdtx\" (UniqueName: \"kubernetes.io/projected/d7867e74-3d84-4780-88b8-a78222af9a95-kube-api-access-trdtx\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.319967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.319930 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-metrics-certs\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.320165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.319990 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7867e74-3d84-4780-88b8-a78222af9a95-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.320165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.320044 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-maas-api-tls\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.320165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.320068 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trdtx\" (UniqueName: \"kubernetes.io/projected/d7867e74-3d84-4780-88b8-a78222af9a95-kube-api-access-trdtx\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.320744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.320721 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7867e74-3d84-4780-88b8-a78222af9a95-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.322735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.322713 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-metrics-certs\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.322863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.322771 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-maas-api-tls\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.338133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.338084 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trdtx\" (UniqueName: \"kubernetes.io/projected/d7867e74-3d84-4780-88b8-a78222af9a95-kube-api-access-trdtx\") pod \"maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.368485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.368459 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:08.390438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.390413 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:08.689022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:08.688991 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk"] Oct 30 03:42:08.689329 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:42:08.689308 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7867e74_3d84_4780_88b8_a78222af9a95.slice/crio-70738a5417d708479c36deb52445e7959c8e6707c36440807a27995c67f064d4 WatchSource:0}: Error finding container 70738a5417d708479c36deb52445e7959c8e6707c36440807a27995c67f064d4: Status 404 returned error can't find the container with id 70738a5417d708479c36deb52445e7959c8e6707c36440807a27995c67f064d4 Oct 30 03:42:09.458604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:09.458522 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" event={"ID":"d7867e74-3d84-4780-88b8-a78222af9a95","Type":"ContainerStarted","Data":"e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8"} Oct 30 03:42:09.458604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:09.458589 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:09.459050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:09.458611 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" event={"ID":"d7867e74-3d84-4780-88b8-a78222af9a95","Type":"ContainerStarted","Data":"70738a5417d708479c36deb52445e7959c8e6707c36440807a27995c67f064d4"} Oct 30 03:42:09.501832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:09.497301 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" podStartSLOduration=1.087457285 podStartE2EDuration="1.497282284s" totalImagesPullingTime="409.824999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:42:09.495479715 +0000 UTC m=+2353.497832913" watchObservedRunningTime="2026-10-30 03:42:09.497282284 +0000 UTC m=+2353.499635481" Oct 30 03:42:09.607966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:09.607935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:11.186014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:11.185977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:12.648397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:12.648368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:14.035705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:14.035663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:15.388412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:15.388380 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:15.471322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:15.471291 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 03:42:15.483068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:15.483017 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:15.801355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:15.801316 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:16.436486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:16.436455 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 03:42:17.108922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:17.108866 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:18.488301 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:18.488270 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:19.858853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:19.858824 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:20.869962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:20.869930 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:21.301105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:21.301071 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:22.896119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:22.896073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:24.379664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:24.379628 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:24.861479 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:42:24.861444 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:42:25.892534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:25.892498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:27.233733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:27.233699 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:28.704328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:28.704287 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:30.208098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:30.208064 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:31.666689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:31.666655 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:33.145206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:33.145177 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:34.677293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:34.677260 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:34.778490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:34.778456 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:34.878050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:34.878018 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:36.300713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:36.300676 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:37.647177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:37.647145 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:39.150438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:39.150322 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:40.188661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:40.188627 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:42:40.592703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:40.592670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:42.085484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:42.085444 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:43.585286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:43.585249 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:44.007722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:44.007692 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:44.108125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:44.108093 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:44.934835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:44.934802 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:45.047498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:45.047388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:45.266264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:45.266224 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:46.810909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:46.810871 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:46.911128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:46.911101 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:48.104186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:48.104146 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:48.198924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:48.198895 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:48.307408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:48.307376 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:48.408508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:48.408438 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:48.697370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:48.697292 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:48.833460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:48.833430 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:48.933099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:48.933072 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:49.040483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:49.040452 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:49.467327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:49.467240 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:49.568567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:49.568529 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:49.982934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:49.982900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:50.086207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:50.086178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:50.858910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:50.858877 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:51.115126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:51.115041 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:51.707068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:51.707031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:53.086769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:53.086727 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:54.479338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:54.479310 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:54.860992 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:42:54.860955 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:42:55.058234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:55.058203 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:55.060249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:55.060226 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh"] Oct 30 03:42:55.155950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:55.155840 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:55.692872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:55.692830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:56.059892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:56.059859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:56.463311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:56.463237 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:42:56.467229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:56.467205 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:42:56.804739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:56.804678 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:56.908411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:56.908380 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:57.439639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:57.439605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:57.596399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:57.596362 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:58.084555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:58.084521 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:58.949120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:58.949082 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:59.048512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:59.048470 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:42:59.208521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:42:59.208439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:00.060919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.060862 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" podUID="fc5cea7f-a393-4170-a641-af2768a33cca" containerName="payload-processing" containerID="cri-o://3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39" gracePeriod=25 Oct 30 03:43:00.325390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.325334 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:43:00.503333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.503304 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-plugins-config-volume\") pod \"fc5cea7f-a393-4170-a641-af2768a33cca\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " Oct 30 03:43:00.503513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.503341 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-routing-overlay-volume\") pod \"fc5cea7f-a393-4170-a641-af2768a33cca\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " Oct 30 03:43:00.503513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.503404 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-kube-api-access-sh8b9\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-kube-api-access-sh8b9\") pod \"fc5cea7f-a393-4170-a641-af2768a33cca\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " Oct 30 03:43:00.503513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.503424 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-provider-credentials\") pod \"fc5cea7f-a393-4170-a641-af2768a33cca\" (UID: \"fc5cea7f-a393-4170-a641-af2768a33cca\") " Oct 30 03:43:00.503837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.503799 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-plugins-config-volume" pod "fc5cea7f-a393-4170-a641-af2768a33cca" (UID: "fc5cea7f-a393-4170-a641-af2768a33cca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:00.503837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.503796 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-routing-overlay-volume" pod "fc5cea7f-a393-4170-a641-af2768a33cca" (UID: "fc5cea7f-a393-4170-a641-af2768a33cca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:00.504146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.504127 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:00.504235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.504153 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc5cea7f-a393-4170-a641-af2768a33cca-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:00.505966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.505936 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-kube-api-access-sh8b9" pod "fc5cea7f-a393-4170-a641-af2768a33cca" (UID: "fc5cea7f-a393-4170-a641-af2768a33cca"). InnerVolumeSpecName "kube-api-access-sh8b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:00.505966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.505951 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-provider-credentials" pod "fc5cea7f-a393-4170-a641-af2768a33cca" (UID: "fc5cea7f-a393-4170-a641-af2768a33cca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:00.605231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.605163 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh8b9\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-kube-api-access-sh8b9\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:00.605231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.605189 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc5cea7f-a393-4170-a641-af2768a33cca-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:00.610076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.610051 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:00.818614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.818577 4442 generic.go:356] "Generic (PLEG): container finished" podID="fc5cea7f-a393-4170-a641-af2768a33cca" containerID="3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39" exitCode=0 Oct 30 03:43:00.818838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.818619 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" Oct 30 03:43:00.818838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.818668 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" event={"ID":"fc5cea7f-a393-4170-a641-af2768a33cca","Type":"ContainerDied","Data":"3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39"} Oct 30 03:43:00.818838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.818726 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh" event={"ID":"fc5cea7f-a393-4170-a641-af2768a33cca","Type":"ContainerDied","Data":"5229d4be16edcb1faba6dd2176174ef675f57663dc11c7ea778de741d8156cf5"} Oct 30 03:43:00.818838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.818778 4442 scope.go:122] "RemoveContainer" containerID="3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39" Oct 30 03:43:00.829547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.829530 4442 scope.go:122] "RemoveContainer" containerID="3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39" Oct 30 03:43:00.829877 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:43:00.829850 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39\": container with ID starting with 3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39 not found: ID does not exist" containerID="3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39" Oct 30 03:43:00.829941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.829885 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39"} err="failed to get container status \"3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39\": rpc error: code = NotFound desc = could not find container \"3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39\": container with ID starting with 3bc0a6fe1562abbd88780035d9528031768dcc610134a56fd197481fddc69b39 not found: ID does not exist" Oct 30 03:43:00.868406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.868359 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh"] Oct 30 03:43:00.872308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:00.872289 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-93b821/payload-processing-external-model-e2e-extmod-b-93b821-6f5fqzjmh"] Oct 30 03:43:02.108538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:02.108496 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5cea7f-a393-4170-a641-af2768a33cca" path="/var/lib/kubelet/pods/fc5cea7f-a393-4170-a641-af2768a33cca/volumes" Oct 30 03:43:02.111336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:02.111310 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:02.212628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:02.212597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:02.311893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:02.311859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:03.716536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:03.716498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:03.895415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:03.895383 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:03.995822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:03.995714 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:04.095903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:04.095873 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:04.235308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:04.235275 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:05.091534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:05.091491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:06.614285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:06.614243 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:08.298599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:08.298563 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:08.601939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:08.601836 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:08.702718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:08.702687 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:09.137231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:09.137198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:09.447723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:09.447643 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:09.984306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:09.984277 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:10.362825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.362788 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:10.619363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.619288 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl"] Oct 30 03:43:10.619946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.619929 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fc5cea7f-a393-4170-a641-af2768a33cca" containerName="payload-processing" Oct 30 03:43:10.619946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.619946 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5cea7f-a393-4170-a641-af2768a33cca" containerName="payload-processing" Oct 30 03:43:10.620074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.620069 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5cea7f-a393-4170-a641-af2768a33cca" containerName="payload-processing" Oct 30 03:43:10.623123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.623102 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.640157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.640130 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl"] Oct 30 03:43:10.647848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.647818 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"routing-overlay\"" Oct 30 03:43:10.647848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.647836 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"openshift-service-ca.crt\"" Oct 30 03:43:10.648041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.647919 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"e2e-cmp-legacy-c782a6-provider-cred\"" Oct 30 03:43:10.648041 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.647921 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"payload-processing-external-model-plugins-e2e-extmod-a-9d8692\"" Oct 30 03:43:10.648837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.648813 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"kube-root-ca.crt\"" Oct 30 03:43:10.650530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.650507 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"payload-processing-external-model-e2e--51bfd021-dockercfg-c6vmr\"" Oct 30 03:43:10.682104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.682078 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.682250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.682205 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzh9n\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-kube-api-access-rzh9n\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.682294 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.682258 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.682369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.682357 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.789511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.789461 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.791004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.790976 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.791324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.791304 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzh9n\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-kube-api-access-rzh9n\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.791471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.791454 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.792376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.792353 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.792810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.790669 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.819606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.819556 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.830577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.830504 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzh9n\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-kube-api-access-rzh9n\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.877943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.877851 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:10.936618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.936588 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:10.976916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:10.976885 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:11.243355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.243325 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl"] Oct 30 03:43:11.244790 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:43:11.244735 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9059d996_d059_408b_abdb_e9a76a8f5d69.slice/crio-44ccb683c0d09079dc1d50fdc66071347e7227713c71dcdfc517610982f3f630 WatchSource:0}: Error finding container 44ccb683c0d09079dc1d50fdc66071347e7227713c71dcdfc517610982f3f630: Status 404 returned error can't find the container with id 44ccb683c0d09079dc1d50fdc66071347e7227713c71dcdfc517610982f3f630 Oct 30 03:43:11.423777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.423459 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:11.539244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.539213 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:11.716971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.716932 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:11.908782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.908658 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" event={"ID":"9059d996-d059-408b-abdb-e9a76a8f5d69","Type":"ContainerStarted","Data":"f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7"} Oct 30 03:43:11.908782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.908721 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" event={"ID":"9059d996-d059-408b-abdb-e9a76a8f5d69","Type":"ContainerStarted","Data":"44ccb683c0d09079dc1d50fdc66071347e7227713c71dcdfc517610982f3f630"} Oct 30 03:43:11.909046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.909027 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:11.945361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:11.945316 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" podStartSLOduration=1.945302721 podStartE2EDuration="1.945302721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:11.943625229 +0000 UTC m=+2415.945978449" watchObservedRunningTime="2026-10-30 03:43:11.945302721 +0000 UTC m=+2415.947655917" Oct 30 03:43:12.269097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:12.269059 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:13.377368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:13.377336 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:14.067247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.067202 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc"] Oct 30 03:43:14.071518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.071497 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.084745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.084723 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-855e14-openshift-default-dockercfg-24jzw\"" Oct 30 03:43:14.098785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.098741 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc"] Oct 30 03:43:14.236716 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.236680 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95105ce-6081-4317-8aaf-c38e36b7d770-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.236922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.236726 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.236922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.236819 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.236922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.236903 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.237106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.236952 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.237165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.237127 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.237221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.237165 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ch6\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-kube-api-access-c2ch6\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.237284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.237269 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.237335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.237293 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338179 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338225 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ch6\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-kube-api-access-c2ch6\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338277 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338303 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338367 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95105ce-6081-4317-8aaf-c38e36b7d770-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338407 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338449 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338479 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338515 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338578 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338653 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-data\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.338831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338815 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-credential-socket\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.339037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.338868 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-certs\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.339166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.339141 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95105ce-6081-4317-8aaf-c38e36b7d770-istiod-ca-cert\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.340917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.340892 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-envoy\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.341036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.341019 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-podinfo\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.356389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.356359 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-token\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.356529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.356512 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ch6\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-kube-api-access-c2ch6\") pod \"e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.357803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.357746 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:14.385785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.385743 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:14.728338 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:43:14.728294 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95105ce_6081_4317_8aaf_c38e36b7d770.slice/crio-e84ce35271a5c9fa818b32e8a9f836844f9a53e0e10bfcb1352d59bcb43ea600 WatchSource:0}: Error finding container e84ce35271a5c9fa818b32e8a9f836844f9a53e0e10bfcb1352d59bcb43ea600: Status 404 returned error can't find the container with id e84ce35271a5c9fa818b32e8a9f836844f9a53e0e10bfcb1352d59bcb43ea600 Oct 30 03:43:14.728500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.728469 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc"] Oct 30 03:43:14.732141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.732108 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:43:14.732255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.732175 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:43:14.732255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.732206 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:43:14.931001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.930970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" event={"ID":"f95105ce-6081-4317-8aaf-c38e36b7d770","Type":"ContainerStarted","Data":"178175321abe2835ddbf5d9009ef9f654806a977df6b3f87ad7bc38f25f7cad3"} Oct 30 03:43:14.931205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.931018 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" event={"ID":"f95105ce-6081-4317-8aaf-c38e36b7d770","Type":"ContainerStarted","Data":"e84ce35271a5c9fa818b32e8a9f836844f9a53e0e10bfcb1352d59bcb43ea600"} Oct 30 03:43:14.972128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.972073 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podStartSLOduration=0.972058797 podStartE2EDuration="972.058797ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:14.968090419 +0000 UTC m=+2418.970443616" watchObservedRunningTime="2026-10-30 03:43:14.972058797 +0000 UTC m=+2418.974411993" Oct 30 03:43:14.980886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:14.980813 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:15.386522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:15.386494 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:15.387627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:15.387604 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 03:43:15.387727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:15.387657 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 03:43:15.629132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:15.629095 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:15.726210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:15.726123 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:15.902779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:15.897292 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:16.208363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:16.208318 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:16.355023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:16.354985 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:16.388780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:16.387151 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 03:43:16.388780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:16.387211 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 03:43:16.457904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:16.457872 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:16.554885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:16.554858 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:17.019930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:17.019896 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:17.117922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:17.117882 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:17.389331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:17.389229 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 03:43:17.389987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:17.389950 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 03:43:18.264908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:18.264876 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:18.388777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:18.388213 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 03:43:18.388777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:18.388272 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 03:43:18.512845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:18.512810 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:19.397191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:19.397156 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 03:43:19.397472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:19.397446 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 03:43:19.738379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:19.738294 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:20.404781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:20.404133 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Oct 30 03:43:20.404781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:20.404195 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Oct 30 03:43:21.337642 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:21.337605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:21.595344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:21.595256 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:21.596527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:21.596501 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:21.610879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:21.610853 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:43:22.917652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:22.917619 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:43:22.992636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:22.992590 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:24.371431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:24.371400 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:24.862538 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:43:24.862502 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:43:25.843422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:25.843391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:27.093688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.093656 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj"] Oct 30 03:43:27.094170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.093898 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" podUID="307897d1-e14b-466c-ad08-97e765eae743" containerName="maas-api" containerID="cri-o://42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7" gracePeriod=30 Oct 30 03:43:27.140145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.140117 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:27.500004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.499971 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:43:27.609298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.609257 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/307897d1-e14b-466c-ad08-97e765eae743-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307897d1-e14b-466c-ad08-97e765eae743-openshift-service-ca\") pod \"307897d1-e14b-466c-ad08-97e765eae743\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " Oct 30 03:43:27.609506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.609407 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-maas-api-tls\") pod \"307897d1-e14b-466c-ad08-97e765eae743\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " Oct 30 03:43:27.609506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.609493 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-metrics-certs\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-metrics-certs\") pod \"307897d1-e14b-466c-ad08-97e765eae743\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " Oct 30 03:43:27.609663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.609573 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/307897d1-e14b-466c-ad08-97e765eae743-kube-api-access-w4mnk\" (UniqueName: \"kubernetes.io/projected/307897d1-e14b-466c-ad08-97e765eae743-kube-api-access-w4mnk\") pod \"307897d1-e14b-466c-ad08-97e765eae743\" (UID: \"307897d1-e14b-466c-ad08-97e765eae743\") " Oct 30 03:43:27.610183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.610151 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/307897d1-e14b-466c-ad08-97e765eae743-openshift-service-ca" pod "307897d1-e14b-466c-ad08-97e765eae743" (UID: "307897d1-e14b-466c-ad08-97e765eae743"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:27.612266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.612225 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-maas-api-tls" pod "307897d1-e14b-466c-ad08-97e765eae743" (UID: "307897d1-e14b-466c-ad08-97e765eae743"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:27.614934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.614904 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/307897d1-e14b-466c-ad08-97e765eae743-kube-api-access-w4mnk" pod "307897d1-e14b-466c-ad08-97e765eae743" (UID: "307897d1-e14b-466c-ad08-97e765eae743"). InnerVolumeSpecName "kube-api-access-w4mnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:27.615024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.614988 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-metrics-certs" pod "307897d1-e14b-466c-ad08-97e765eae743" (UID: "307897d1-e14b-466c-ad08-97e765eae743"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:43:27.711881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.711809 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:27.711881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.711835 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/307897d1-e14b-466c-ad08-97e765eae743-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:27.711881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.711846 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4mnk\" (UniqueName: \"kubernetes.io/projected/307897d1-e14b-466c-ad08-97e765eae743-kube-api-access-w4mnk\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:27.711881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.711854 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/307897d1-e14b-466c-ad08-97e765eae743-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:27.881815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:27.881781 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:28.063768 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.063723 4442 generic.go:356] "Generic (PLEG): container finished" podID="307897d1-e14b-466c-ad08-97e765eae743" containerID="42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7" exitCode=0 Oct 30 03:43:28.063946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.063790 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" Oct 30 03:43:28.063946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.063785 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" event={"ID":"307897d1-e14b-466c-ad08-97e765eae743","Type":"ContainerDied","Data":"42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7"} Oct 30 03:43:28.063946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.063915 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj" event={"ID":"307897d1-e14b-466c-ad08-97e765eae743","Type":"ContainerDied","Data":"c7cdc1f2cd5b556174b2637bd8390c50186adc533d66a76194abdff21a63f70f"} Oct 30 03:43:28.063946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.063943 4442 scope.go:122] "RemoveContainer" containerID="42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7" Oct 30 03:43:28.074542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.074511 4442 scope.go:122] "RemoveContainer" containerID="42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7" Oct 30 03:43:28.074821 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:43:28.074802 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7\": container with ID starting with 42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7 not found: ID does not exist" containerID="42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7" Oct 30 03:43:28.074888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.074830 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7"} err="failed to get container status \"42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7\": rpc error: code = NotFound desc = could not find container \"42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7\": container with ID starting with 42945c1eabb00c152ed8014aa19fff13285772c5a7d0e66ff152b87a194affd7 not found: ID does not exist" Oct 30 03:43:28.110370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.110344 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj"] Oct 30 03:43:28.120824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.120805 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-dbed6d-549895c8fd-nj7wj"] Oct 30 03:43:28.395641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.395551 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:28.511892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:28.511845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:29.144987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:29.144948 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:29.514236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:29.514202 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j"] Oct 30 03:43:29.514513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:29.514479 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" containerID="cri-o://fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581" gracePeriod=30 Oct 30 03:43:29.630581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:29.629278 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:29.737597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:29.737565 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:30.107076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:30.107047 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="307897d1-e14b-466c-ad08-97e765eae743" path="/var/lib/kubelet/pods/307897d1-e14b-466c-ad08-97e765eae743/volumes" Oct 30 03:43:31.127104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:31.127078 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:31.595108 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:31.595037 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:32.029085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.029052 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv"] Oct 30 03:43:32.030021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.030001 4442 cpu_manager.go:414] "containerMap: removing container" podUID="307897d1-e14b-466c-ad08-97e765eae743" containerName="maas-api" Oct 30 03:43:32.030081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.030024 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="307897d1-e14b-466c-ad08-97e765eae743" containerName="maas-api" Oct 30 03:43:32.030241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.030230 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="307897d1-e14b-466c-ad08-97e765eae743" containerName="maas-api" Oct 30 03:43:32.035875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.035851 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.055920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.055885 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv"] Oct 30 03:43:32.061825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.061801 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-441461-openshift-default-dockercfg-6dm96\"" Oct 30 03:43:32.069947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.069918 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:32.178865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.178832 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-certs\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.178899 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnkl\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-kube-api-access-jfnkl\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.179054 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-data\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.179125 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-token\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.179172 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-socket\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.179208 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-podinfo\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.179249 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-credential-socket\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.179368 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istiod-ca-cert\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.179599 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.179403 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-envoy\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.181385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.181365 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.283595 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istiod-ca-cert\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.283648 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-envoy\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.283773 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-certs\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.283821 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnkl\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-kube-api-access-jfnkl\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.283904 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-data\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.283962 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-token\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.284001 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-socket\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.284028 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-podinfo\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.284066 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-credential-socket\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.284521 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-credential-socket\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.285609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.285258 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istiod-ca-cert\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.286718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.286696 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-data\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.287187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.287164 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-certs\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.289542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.289512 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-socket\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.320111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.317263 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-envoy\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.320111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.318230 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-podinfo\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.320111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.319447 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnkl\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-kube-api-access-jfnkl\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.330298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.330260 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-token\") pod \"e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.360779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.357352 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:32.686082 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.686045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:32.687355 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:43:32.687329 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388c5dc2_6e38_4f8d_a4b4_6fdf5e61951f.slice/crio-ecd2f8143786ddf6883d541172c88fdefa1d5af90ce412256533638ed2191028 WatchSource:0}: Error finding container ecd2f8143786ddf6883d541172c88fdefa1d5af90ce412256533638ed2191028: Status 404 returned error can't find the container with id ecd2f8143786ddf6883d541172c88fdefa1d5af90ce412256533638ed2191028 Oct 30 03:43:32.687857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.687833 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv"] Oct 30 03:43:32.689983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.689956 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:43:32.690048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.690019 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:43:32.690090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:32.690046 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:43:33.098440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:33.098170 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" event={"ID":"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f","Type":"ContainerStarted","Data":"17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c"} Oct 30 03:43:33.098440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:33.098236 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" event={"ID":"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f","Type":"ContainerStarted","Data":"ecd2f8143786ddf6883d541172c88fdefa1d5af90ce412256533638ed2191028"} Oct 30 03:43:33.153657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:33.153602 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" podStartSLOduration=1.153589462 podStartE2EDuration="1.153589462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:33.149660302 +0000 UTC m=+2437.152013503" watchObservedRunningTime="2026-10-30 03:43:33.153589462 +0000 UTC m=+2437.155942720" Oct 30 03:43:33.358526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:33.358438 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:33.364977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:33.364953 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:33.597867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:33.597829 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:43:34.109352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:34.109318 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:34.109535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:34.109384 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:43:34.385448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:34.385370 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:34.915655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:34.915598 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:43:35.028712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.028679 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-certs\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.028928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.028778 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/643fe397-ece5-48d2-96cf-a088d3856eb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643fe397-ece5-48d2-96cf-a088d3856eb3-istiod-ca-cert\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.028928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.028881 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-envoy\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.029064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.028971 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-token\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-token\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.029064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.028970 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-certs" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:43:35.029064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029022 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-credential-socket\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.029235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029072 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-socket\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.029235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029108 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-kube-api-access-9cp67\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-kube-api-access-9cp67\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.029235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029152 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-podinfo\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.029235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029213 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/643fe397-ece5-48d2-96cf-a088d3856eb3-istiod-ca-cert" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:43:35.029235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029226 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-data\") pod \"643fe397-ece5-48d2-96cf-a088d3856eb3\" (UID: \"643fe397-ece5-48d2-96cf-a088d3856eb3\") " Oct 30 03:43:35.029501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029415 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-socket" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:43:35.029563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029507 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-credential-socket" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:43:35.029645 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029625 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-data" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:43:35.029886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029801 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.029886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029823 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.029886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029832 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.029886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029840 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.029886 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.029850 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/643fe397-ece5-48d2-96cf-a088d3856eb3-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.031959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.031936 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-envoy" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:43:35.032079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.032003 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-kube-api-access-9cp67" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "kube-api-access-9cp67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:35.032079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.032010 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-token" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:43:35.032241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.032222 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-podinfo" pod "643fe397-ece5-48d2-96cf-a088d3856eb3" (UID: "643fe397-ece5-48d2-96cf-a088d3856eb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:43:35.117839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.117802 4442 generic.go:356] "Generic (PLEG): container finished" podID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerID="fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581" exitCode=0 Oct 30 03:43:35.118051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.117852 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" Oct 30 03:43:35.118051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.117932 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" event={"ID":"643fe397-ece5-48d2-96cf-a088d3856eb3","Type":"ContainerDied","Data":"fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581"} Oct 30 03:43:35.118051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.117980 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j" event={"ID":"643fe397-ece5-48d2-96cf-a088d3856eb3","Type":"ContainerDied","Data":"cddea890cb3d594fef0a0685e8070744742faccb3d199c72271b6ddbe8b8ee0e"} Oct 30 03:43:35.118051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.118011 4442 scope.go:122] "RemoveContainer" containerID="fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581" Oct 30 03:43:35.157940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.148090 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.157940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.148122 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.157940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.148137 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cp67\" (UniqueName: \"kubernetes.io/projected/643fe397-ece5-48d2-96cf-a088d3856eb3-kube-api-access-9cp67\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.157940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.148153 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/643fe397-ece5-48d2-96cf-a088d3856eb3-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:43:35.184130 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.184108 4442 scope.go:122] "RemoveContainer" containerID="ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b" Oct 30 03:43:35.194777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.189802 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j"] Oct 30 03:43:35.196782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.195988 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-dbed6d-openshift-default-7bcf6b6bdf-bvq6j"] Oct 30 03:43:35.216879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.216848 4442 scope.go:122] "RemoveContainer" containerID="fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581" Oct 30 03:43:35.217359 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:43:35.217335 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581\": container with ID starting with fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581 not found: ID does not exist" containerID="fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581" Oct 30 03:43:35.217491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.217369 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581"} err="failed to get container status \"fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581\": rpc error: code = NotFound desc = could not find container \"fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581\": container with ID starting with fdaf05025a043ab5f342520392074ebaba35173b5baec6498b357fa48b315581 not found: ID does not exist" Oct 30 03:43:35.217491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.217396 4442 scope.go:122] "RemoveContainer" containerID="ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b" Oct 30 03:43:35.217769 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:43:35.217730 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b\": container with ID starting with ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b not found: ID does not exist" containerID="ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b" Oct 30 03:43:35.217849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.217786 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b"} err="failed to get container status \"ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b\": rpc error: code = NotFound desc = could not find container \"ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b\": container with ID starting with ea756155cb60dea54d3bae3edfa1389ccbdc35ba8f973518517329decc82c48b not found: ID does not exist" Oct 30 03:43:35.539905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.539868 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:35.643480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:35.643451 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:36.108262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:36.108229 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" path="/var/lib/kubelet/pods/643fe397-ece5-48d2-96cf-a088d3856eb3/volumes" Oct 30 03:43:36.200731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:36.200701 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:37.201178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.201143 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:37.302874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.302839 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:37.327060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.327030 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln"] Oct 30 03:43:37.327710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.327696 4442 cpu_manager.go:414] "containerMap: removing container" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" Oct 30 03:43:37.327791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.327712 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" Oct 30 03:43:37.327791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.327721 4442 cpu_manager.go:414] "containerMap: removing container" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" Oct 30 03:43:37.327791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.327726 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" Oct 30 03:43:37.327950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.327831 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" Oct 30 03:43:37.327950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.327840 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="643fe397-ece5-48d2-96cf-a088d3856eb3" containerName="istio-proxy" Oct 30 03:43:37.332814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.332796 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.349633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.349611 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln"] Oct 30 03:43:37.356493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.356470 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-441461\"" Oct 30 03:43:37.356623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.356478 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-441461-dockercfg-j95hr\"" Oct 30 03:43:37.392821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.392795 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb2df\" (UniqueName: \"kubernetes.io/projected/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-kube-api-access-jb2df\") pod \"payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.393004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.392935 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.396390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.396370 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s"] Oct 30 03:43:37.400467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.400453 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:37.417650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.417631 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s"] Oct 30 03:43:37.528493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.517460 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.528493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.517599 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb2df\" (UniqueName: \"kubernetes.io/projected/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-kube-api-access-jb2df\") pod \"payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.528493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.518907 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.570554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.570510 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb2df\" (UniqueName: \"kubernetes.io/projected/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-kube-api-access-jb2df\") pod \"payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.618965 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.618932 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:37.619147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.619126 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9ct\" (UniqueName: \"kubernetes.io/projected/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-kube-api-access-wk9ct\") pod \"payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:37.646672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.646637 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:37.720276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.720236 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9ct\" (UniqueName: \"kubernetes.io/projected/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-kube-api-access-wk9ct\") pod \"payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:37.720440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.720381 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:37.721791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.721327 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:37.741680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.741652 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9ct\" (UniqueName: \"kubernetes.io/projected/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-kube-api-access-wk9ct\") pod \"payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:37.939178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.939096 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:37.953797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:37.950024 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln"] Oct 30 03:43:38.020780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:38.017985 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:38.139612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:38.139576 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" event={"ID":"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e","Type":"ContainerStarted","Data":"a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65"} Oct 30 03:43:38.139813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:38.139640 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" event={"ID":"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e","Type":"ContainerStarted","Data":"f810fa7d65d55c5223541d8ab04bddf90bd3877f57b7fe4494f5da597e24fad7"} Oct 30 03:43:38.139813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:38.139668 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:38.177321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:38.177273 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" podStartSLOduration=1.177258655 podStartE2EDuration="1.177258655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:38.174952655 +0000 UTC m=+2442.177305853" watchObservedRunningTime="2026-10-30 03:43:38.177258655 +0000 UTC m=+2442.179611851" Oct 30 03:43:38.356121 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:38.356095 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s"] Oct 30 03:43:38.356727 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:43:38.356699 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5d678d_d1c0_4f80_bbb6_2ee5af311d78.slice/crio-5b697b39787a5fa1a8a3a172a6786a97309537cd4ec4c9bce8595ec47ab44a0d WatchSource:0}: Error finding container 5b697b39787a5fa1a8a3a172a6786a97309537cd4ec4c9bce8595ec47ab44a0d: Status 404 returned error can't find the container with id 5b697b39787a5fa1a8a3a172a6786a97309537cd4ec4c9bce8595ec47ab44a0d Oct 30 03:43:39.148347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:39.148310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" event={"ID":"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78","Type":"ContainerStarted","Data":"3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553"} Oct 30 03:43:39.148551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:39.148371 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:39.148551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:39.148394 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" event={"ID":"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78","Type":"ContainerStarted","Data":"5b697b39787a5fa1a8a3a172a6786a97309537cd4ec4c9bce8595ec47ab44a0d"} Oct 30 03:43:39.185863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:39.185808 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" podStartSLOduration=2.185794217 podStartE2EDuration="2.185794217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:39.184454037 +0000 UTC m=+2443.186807238" watchObservedRunningTime="2026-10-30 03:43:39.185794217 +0000 UTC m=+2443.188147413" Oct 30 03:43:39.479623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:39.479548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:40.983254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:40.983216 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:42.505042 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:42.505011 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:43.974001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:43.973962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:45.481986 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:45.481956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:47.117920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:47.117886 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:48.711904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:48.711868 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:49.151521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:49.151496 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 03:43:49.176324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:49.176298 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:50.157601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:50.157575 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 03:43:50.196426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:50.196399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:50.310568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:50.310539 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:51.233475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:51.233440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:51.841024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:51.840990 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:53.314699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:53.314665 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:54.862415 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:43:54.862379 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:43:54.983282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:54.983249 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:56.430197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:56.430129 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:58.007976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:58.007941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:43:59.624094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:43:59.624056 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:01.273435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:01.273398 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:02.817721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:02.817685 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:02.962455 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:02.962425 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:44:04.260411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:04.260375 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:05.720555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:05.720515 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:07.315091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:07.315037 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:07.643382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:07.643293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:07.744036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:07.744003 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:09.247060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:09.246970 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:09.363332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:09.363295 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:09.448351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:09.448324 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:10.729494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:10.729463 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:10.833948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:10.833919 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:11.009367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.009336 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v"] Oct 30 03:44:11.016074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.016052 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.029368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.029342 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"agc-cred-5e51f708321f\"" Oct 30 03:44:11.032168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.032144 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v"] Oct 30 03:44:11.039483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.039398 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e10c0f\"" Oct 30 03:44:11.039483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.039413 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"openshift-service-ca.crt\"" Oct 30 03:44:11.039663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.039524 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"routing-overlay\"" Oct 30 03:44:11.040295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.040260 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"kube-root-ca.crt\"" Oct 30 03:44:11.040407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.040308 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-e2e--4f8d4144-dockercfg-ztxfq\"" Oct 30 03:44:11.153711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.153657 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.154037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.154012 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.154317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.154297 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.154611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.154591 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp82\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-kube-api-access-gvp82\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.256187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.256151 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.256369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.256233 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp82\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-kube-api-access-gvp82\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.256369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.256347 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.256473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.256391 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.257184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.257147 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.257308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.257223 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.259007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.258987 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.273950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.273826 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp82\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-kube-api-access-gvp82\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.329292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.329267 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:11.631737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:11.631703 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v"] Oct 30 03:44:11.631906 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:44:11.631875 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode222eca2_a7b8_4f32_806a_bdc52a2b9691.slice/crio-a73e8c767752e4eda42343270689ef2a0b9cc339f8d7d4f1b2d1564e89b75392 WatchSource:0}: Error finding container a73e8c767752e4eda42343270689ef2a0b9cc339f8d7d4f1b2d1564e89b75392: Status 404 returned error can't find the container with id a73e8c767752e4eda42343270689ef2a0b9cc339f8d7d4f1b2d1564e89b75392 Oct 30 03:44:12.376592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:12.376556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:12.410967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:12.410931 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" event={"ID":"e222eca2-a7b8-4f32-806a-bdc52a2b9691","Type":"ContainerStarted","Data":"d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72"} Oct 30 03:44:12.411113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:12.410989 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" event={"ID":"e222eca2-a7b8-4f32-806a-bdc52a2b9691","Type":"ContainerStarted","Data":"a73e8c767752e4eda42343270689ef2a0b9cc339f8d7d4f1b2d1564e89b75392"} Oct 30 03:44:12.411180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:12.411157 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:13.866663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:13.866613 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:15.434202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:15.434172 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:15.543858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:15.543803 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" podStartSLOduration=5.543788041 podStartE2EDuration="5.543788041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:12.467265874 +0000 UTC m=+2476.469619072" watchObservedRunningTime="2026-10-30 03:44:15.543788041 +0000 UTC m=+2479.546141238" Oct 30 03:44:15.544505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:15.544482 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v"] Oct 30 03:44:15.610785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:15.610716 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:16.990007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:16.989934 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:18.467052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:18.466997 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:19.887938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:19.887902 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:20.545156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:20.545021 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" podUID="e222eca2-a7b8-4f32-806a-bdc52a2b9691" containerName="payload-processing" containerID="cri-o://d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72" gracePeriod=25 Oct 30 03:44:20.997781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:20.986951 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:21.140416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.140314 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-plugins-config-volume\") pod \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " Oct 30 03:44:21.140607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.140425 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-routing-overlay-volume\") pod \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " Oct 30 03:44:21.140607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.140468 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-kube-api-access-gvp82\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-kube-api-access-gvp82\") pod \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " Oct 30 03:44:21.140607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.140579 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-provider-credentials\") pod \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\" (UID: \"e222eca2-a7b8-4f32-806a-bdc52a2b9691\") " Oct 30 03:44:21.140792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.140766 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-routing-overlay-volume" pod "e222eca2-a7b8-4f32-806a-bdc52a2b9691" (UID: "e222eca2-a7b8-4f32-806a-bdc52a2b9691"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:21.140843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.140789 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-plugins-config-volume" pod "e222eca2-a7b8-4f32-806a-bdc52a2b9691" (UID: "e222eca2-a7b8-4f32-806a-bdc52a2b9691"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:21.141063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.141048 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:21.141102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.141068 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e222eca2-a7b8-4f32-806a-bdc52a2b9691-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:21.142937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.142917 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-provider-credentials" pod "e222eca2-a7b8-4f32-806a-bdc52a2b9691" (UID: "e222eca2-a7b8-4f32-806a-bdc52a2b9691"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:21.143033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.142939 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-kube-api-access-gvp82" pod "e222eca2-a7b8-4f32-806a-bdc52a2b9691" (UID: "e222eca2-a7b8-4f32-806a-bdc52a2b9691"). InnerVolumeSpecName "kube-api-access-gvp82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:21.242223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.242184 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvp82\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-kube-api-access-gvp82\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:21.242223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.242219 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e222eca2-a7b8-4f32-806a-bdc52a2b9691-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:21.510555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.510516 4442 generic.go:356] "Generic (PLEG): container finished" podID="e222eca2-a7b8-4f32-806a-bdc52a2b9691" containerID="d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72" exitCode=0 Oct 30 03:44:21.510817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.510575 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" Oct 30 03:44:21.510817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.510571 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" event={"ID":"e222eca2-a7b8-4f32-806a-bdc52a2b9691","Type":"ContainerDied","Data":"d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72"} Oct 30 03:44:21.510817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.510712 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v" event={"ID":"e222eca2-a7b8-4f32-806a-bdc52a2b9691","Type":"ContainerDied","Data":"a73e8c767752e4eda42343270689ef2a0b9cc339f8d7d4f1b2d1564e89b75392"} Oct 30 03:44:21.510817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.510740 4442 scope.go:122] "RemoveContainer" containerID="d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72" Oct 30 03:44:21.523681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.523651 4442 scope.go:122] "RemoveContainer" containerID="d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72" Oct 30 03:44:21.524092 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:21.524069 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72\": container with ID starting with d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72 not found: ID does not exist" containerID="d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72" Oct 30 03:44:21.524177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.524101 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72"} err="failed to get container status \"d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72\": rpc error: code = NotFound desc = could not find container \"d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72\": container with ID starting with d8457bd60257ff4d1e5093f04ff6c27179ad27cf706521f54bdf7bb21d6f6d72 not found: ID does not exist" Oct 30 03:44:21.564496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.564473 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v"] Oct 30 03:44:21.567653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.567629 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-b578sqg9v"] Oct 30 03:44:21.608516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:21.608485 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:22.124933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:22.124902 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e222eca2-a7b8-4f32-806a-bdc52a2b9691" path="/var/lib/kubelet/pods/e222eca2-a7b8-4f32-806a-bdc52a2b9691/volumes" Oct 30 03:44:23.373057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:23.373029 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:23.490777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:23.490011 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:24.860324 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:24.860290 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:44:25.005779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:25.004717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:26.537329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:26.537300 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:28.029724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:28.029689 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:29.494096 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:29.494059 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:30.598999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:30.598969 4442 generic.go:356] "Generic (PLEG): container finished" podID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerID="17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c" exitCode=0 Oct 30 03:44:30.599500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:30.599049 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" event={"ID":"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f","Type":"ContainerDied","Data":"17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c"} Oct 30 03:44:30.599500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:30.599409 4442 scope.go:122] "RemoveContainer" containerID="17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c" Oct 30 03:44:30.601044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:30.601017 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:44:30.601145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:30.601084 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:44:30.601145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:30.601115 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:44:30.963664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:30.963587 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:31.613935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:31.613900 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" event={"ID":"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f","Type":"ContainerStarted","Data":"0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62"} Oct 30 03:44:32.358398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:32.358369 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:44:32.359906 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:32.359872 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 03:44:32.360153 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:32.360113 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 03:44:32.408144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:32.408111 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:33.358137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:33.358099 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 03:44:33.358553 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:33.358165 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 03:44:34.108378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.108342 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:34.115652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.115626 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:44:34.357725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.357682 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 03:44:34.357930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.357772 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 03:44:34.559924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.559882 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:34.825028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.824942 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl"] Oct 30 03:44:34.835092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.835049 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:34.936104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:34.936068 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:35.243305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.243210 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:35.358473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.358443 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 03:44:35.358633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.358491 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 03:44:35.372932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.372898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:35.530724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.530690 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:35.705779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.704727 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:35.826339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.826242 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:35.980170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:35.980130 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:36.089009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:36.087465 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:36.358230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:36.358147 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 03:44:36.358230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:36.358200 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 03:44:36.607215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:36.607178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:36.733604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:36.733522 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:37.363484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:37.363456 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:44:37.364462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:37.364439 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 03:44:37.786654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:37.786626 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:38.073350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.073271 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:38.301426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.301393 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9"] Oct 30 03:44:38.302043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.302028 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e222eca2-a7b8-4f32-806a-bdc52a2b9691" containerName="payload-processing" Oct 30 03:44:38.302099 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.302045 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e222eca2-a7b8-4f32-806a-bdc52a2b9691" containerName="payload-processing" Oct 30 03:44:38.302168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.302159 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e222eca2-a7b8-4f32-806a-bdc52a2b9691" containerName="payload-processing" Oct 30 03:44:38.305172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.305152 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.324106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.324045 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"agc-cred-1fa0412344ba\"" Oct 30 03:44:38.324264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.324144 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"payload-processing-external-model-e2e--51bfd021-dockercfg-tc4d6\"" Oct 30 03:44:38.328891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.328865 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9"] Oct 30 03:44:38.449807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.444945 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.449807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.445074 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.449807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.445123 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtgch\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-kube-api-access-qtgch\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.449807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.445152 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.545978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.545946 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.546193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.546003 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtgch\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-kube-api-access-qtgch\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.546193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.546036 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.546337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.546230 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.546918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.546893 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.547148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.547122 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.548971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.548945 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.564120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.564092 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtgch\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-kube-api-access-qtgch\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.618825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.618733 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:38.916652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:38.916627 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9"] Oct 30 03:44:38.917709 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:44:38.917680 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007939d3_624d_4e3f_b50a_7644361df5ae.slice/crio-705056f6ea9b1efc75b5b61154ed90f5128aa4bacbdba7525131614924756528 WatchSource:0}: Error finding container 705056f6ea9b1efc75b5b61154ed90f5128aa4bacbdba7525131614924756528: Status 404 returned error can't find the container with id 705056f6ea9b1efc75b5b61154ed90f5128aa4bacbdba7525131614924756528 Oct 30 03:44:39.160093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.160060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:39.373972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.373939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:39.473190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.473156 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:39.688068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.680967 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:39.715808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.715776 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" event={"ID":"007939d3-624d-4e3f-b50a-7644361df5ae","Type":"ContainerStarted","Data":"9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951"} Oct 30 03:44:39.715978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.715834 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" event={"ID":"007939d3-624d-4e3f-b50a-7644361df5ae","Type":"ContainerStarted","Data":"705056f6ea9b1efc75b5b61154ed90f5128aa4bacbdba7525131614924756528"} Oct 30 03:44:39.715978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.715873 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:39.742484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.742411 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj"] Oct 30 03:44:39.753433 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.753062 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.788945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.782096 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj"] Oct 30 03:44:39.788945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.783100 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:39.788945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.785366 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" podStartSLOduration=1.7853451740000001 podStartE2EDuration="1.785345174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:39.760884132 +0000 UTC m=+2503.763237331" watchObservedRunningTime="2026-10-30 03:44:39.785345174 +0000 UTC m=+2503.787698400" Oct 30 03:44:39.807102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.802023 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"e2e-cfl-85dd43-provider-cred\"" Oct 30 03:44:39.828248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.828209 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" podUID="9059d996-d059-408b-abdb-e9a76a8f5d69" containerName="payload-processing" containerID="cri-o://f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7" gracePeriod=25 Oct 30 03:44:39.884554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.884526 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26tw\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-kube-api-access-v26tw\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.884834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.884808 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.884917 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:39.884871 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:39.884917 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:39.884892 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:39.885034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.884948 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.885034 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:39.884965 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials podName:007939d3-624d-4e3f-b50a-7644361df5ae nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.384941887 +0000 UTC m=+2504.387295089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:39.885205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.885038 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.986971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.986935 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.987437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.987011 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.987437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.987038 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.987437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.987071 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26tw\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-kube-api-access-v26tw\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.987437 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:39.987202 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:39.988233 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:39.987234 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:39.988510 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:39.988498 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials podName:80b700fc-d8d4-45ef-91c6-a189874a0d1f nodeName:}" failed. No retries permitted until 2026-10-30 03:44:40.488449307 +0000 UTC m=+2504.490802485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:39.988641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.988618 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:39.989130 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:39.989107 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:40.005799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.005770 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26tw\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-kube-api-access-v26tw\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:40.108838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.108808 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:44:40.291075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.291042 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-routing-overlay-volume\") pod \"9059d996-d059-408b-abdb-e9a76a8f5d69\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " Oct 30 03:44:40.291263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.291147 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-kube-api-access-rzh9n\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-kube-api-access-rzh9n\") pod \"9059d996-d059-408b-abdb-e9a76a8f5d69\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " Oct 30 03:44:40.291263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.291200 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-provider-credentials\") pod \"9059d996-d059-408b-abdb-e9a76a8f5d69\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " Oct 30 03:44:40.291381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.291290 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-plugins-config-volume\") pod \"9059d996-d059-408b-abdb-e9a76a8f5d69\" (UID: \"9059d996-d059-408b-abdb-e9a76a8f5d69\") " Oct 30 03:44:40.291476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.291453 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-routing-overlay-volume" pod "9059d996-d059-408b-abdb-e9a76a8f5d69" (UID: "9059d996-d059-408b-abdb-e9a76a8f5d69"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:40.291701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.291679 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-plugins-config-volume" pod "9059d996-d059-408b-abdb-e9a76a8f5d69" (UID: "9059d996-d059-408b-abdb-e9a76a8f5d69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:40.292149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.292126 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:40.292219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.292151 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9059d996-d059-408b-abdb-e9a76a8f5d69-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:40.293551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.293533 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-provider-credentials" pod "9059d996-d059-408b-abdb-e9a76a8f5d69" (UID: "9059d996-d059-408b-abdb-e9a76a8f5d69"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:40.293607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.293563 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-kube-api-access-rzh9n" pod "9059d996-d059-408b-abdb-e9a76a8f5d69" (UID: "9059d996-d059-408b-abdb-e9a76a8f5d69"). InnerVolumeSpecName "kube-api-access-rzh9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:40.393777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.393716 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzh9n\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-kube-api-access-rzh9n\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:40.393777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.393742 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9059d996-d059-408b-abdb-e9a76a8f5d69-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:40.393777 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:40.393768 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:40.393777 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:40.393788 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:40.394084 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:40.393867 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials podName:007939d3-624d-4e3f-b50a-7644361df5ae nodeName:}" failed. No retries permitted until 2026-10-30 03:44:41.393839784 +0000 UTC m=+2505.396192984 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:40.495112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.495071 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:40.495275 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:40.495216 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:40.495275 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:40.495238 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:40.495371 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:40.495290 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials podName:80b700fc-d8d4-45ef-91c6-a189874a0d1f nodeName:}" failed. No retries permitted until 2026-10-30 03:44:41.495275903 +0000 UTC m=+2505.497629090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:40.568348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.568318 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9"] Oct 30 03:44:40.595051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.595021 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc"] Oct 30 03:44:40.595651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.595638 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9059d996-d059-408b-abdb-e9a76a8f5d69" containerName="payload-processing" Oct 30 03:44:40.595696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.595652 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9059d996-d059-408b-abdb-e9a76a8f5d69" containerName="payload-processing" Oct 30 03:44:40.595823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.595811 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9059d996-d059-408b-abdb-e9a76a8f5d69" containerName="payload-processing" Oct 30 03:44:40.599650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.599635 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.617317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.617281 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc"] Oct 30 03:44:40.698383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.698343 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7df\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-kube-api-access-ww7df\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.698575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.698557 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.698618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.698595 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.698801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.698779 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.724416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.724385 4442 generic.go:356] "Generic (PLEG): container finished" podID="9059d996-d059-408b-abdb-e9a76a8f5d69" containerID="f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7" exitCode=0 Oct 30 03:44:40.724575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.724432 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" Oct 30 03:44:40.724575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.724469 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" event={"ID":"9059d996-d059-408b-abdb-e9a76a8f5d69","Type":"ContainerDied","Data":"f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7"} Oct 30 03:44:40.724575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.724522 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl" event={"ID":"9059d996-d059-408b-abdb-e9a76a8f5d69","Type":"ContainerDied","Data":"44ccb683c0d09079dc1d50fdc66071347e7227713c71dcdfc517610982f3f630"} Oct 30 03:44:40.724575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.724547 4442 scope.go:122] "RemoveContainer" containerID="f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7" Oct 30 03:44:40.736677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.736659 4442 scope.go:122] "RemoveContainer" containerID="f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7" Oct 30 03:44:40.738671 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:40.737370 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7\": container with ID starting with f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7 not found: ID does not exist" containerID="f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7" Oct 30 03:44:40.738671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.737406 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7"} err="failed to get container status \"f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7\": rpc error: code = NotFound desc = could not find container \"f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7\": container with ID starting with f837bd202476b9bc04b3dc999219a8cc39bc4723d283d18aba89d716190741d7 not found: ID does not exist" Oct 30 03:44:40.777459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.777439 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl"] Oct 30 03:44:40.781122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.781104 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-77f89d5vl"] Oct 30 03:44:40.799935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.799916 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7df\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-kube-api-access-ww7df\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.800081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.800066 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.800133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.800097 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.800175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.800154 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.800692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.800665 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.800823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.800808 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.802616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.802594 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.817841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.817781 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7df\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-kube-api-access-ww7df\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.915535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.915500 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:40.919163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:40.919132 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:41.255900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:41.255873 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc"] Oct 30 03:44:41.256210 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:44:41.255934 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001a8986_c6b5_4613_a492_b5ad66989952.slice/crio-3d9a69c29114b08086bd73d8914fb83f81b25b2fc7a2f43d85c8c186a8ba4cbd WatchSource:0}: Error finding container 3d9a69c29114b08086bd73d8914fb83f81b25b2fc7a2f43d85c8c186a8ba4cbd: Status 404 returned error can't find the container with id 3d9a69c29114b08086bd73d8914fb83f81b25b2fc7a2f43d85c8c186a8ba4cbd Oct 30 03:44:41.410235 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:41.410199 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:41.410235 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:41.410227 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:41.410471 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:41.410277 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials podName:007939d3-624d-4e3f-b50a-7644361df5ae nodeName:}" failed. No retries permitted until 2026-10-30 03:44:43.410263372 +0000 UTC m=+2507.412616547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:41.512012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:41.511915 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:41.512189 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:41.512078 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:41.512189 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:41.512105 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:41.512189 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:41.512169 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials podName:80b700fc-d8d4-45ef-91c6-a189874a0d1f nodeName:}" failed. No retries permitted until 2026-10-30 03:44:43.512150997 +0000 UTC m=+2507.514504172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:41.735554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:41.735516 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" event={"ID":"001a8986-c6b5-4613-a492-b5ad66989952","Type":"ContainerStarted","Data":"17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1"} Oct 30 03:44:41.735709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:41.735575 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" event={"ID":"001a8986-c6b5-4613-a492-b5ad66989952","Type":"ContainerStarted","Data":"3d9a69c29114b08086bd73d8914fb83f81b25b2fc7a2f43d85c8c186a8ba4cbd"} Oct 30 03:44:41.735802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:41.735747 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:41.805603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:41.805504 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" podStartSLOduration=1.805491728 podStartE2EDuration="1.805491728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:41.805308092 +0000 UTC m=+2505.807661306" watchObservedRunningTime="2026-10-30 03:44:41.805491728 +0000 UTC m=+2505.807844925" Oct 30 03:44:42.108108 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:42.108014 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9059d996-d059-408b-abdb-e9a76a8f5d69" path="/var/lib/kubelet/pods/9059d996-d059-408b-abdb-e9a76a8f5d69/volumes" Oct 30 03:44:42.453223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:42.453130 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:43.436153 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:43.436119 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:43.436153 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:43.436142 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:43.436378 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:43.436196 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials podName:007939d3-624d-4e3f-b50a-7644361df5ae nodeName:}" failed. No retries permitted until 2026-10-30 03:44:47.436183392 +0000 UTC m=+2511.438536566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:43.536651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:43.536624 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:43.537030 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:43.536787 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:43.537030 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:43.536809 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:43.537030 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:43.536867 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials podName:80b700fc-d8d4-45ef-91c6-a189874a0d1f nodeName:}" failed. No retries permitted until 2026-10-30 03:44:47.536853714 +0000 UTC m=+2511.539206893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:43.921841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:43.921801 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:45.046665 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:45.046623 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:45.286471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:45.286437 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:45.457368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:45.457284 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:45.558644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:45.558610 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:45.769250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:45.769174 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:46.123943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.123862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:46.249574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.249542 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:46.416954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.416872 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:46.587572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.587543 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:46.685869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.685776 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:46.743892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.743839 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" podUID="007939d3-624d-4e3f-b50a-7644361df5ae" containerName="payload-processing" containerID="cri-o://9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951" gracePeriod=25 Oct 30 03:44:46.879879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.879834 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:46.995670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:46.994446 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:47.098493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.098464 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:47.108331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.108304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:47.196040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196008 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-routing-overlay-volume\") pod \"007939d3-624d-4e3f-b50a-7644361df5ae\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " Oct 30 03:44:47.196467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196050 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials\") pod \"007939d3-624d-4e3f-b50a-7644361df5ae\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " Oct 30 03:44:47.196467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196208 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-plugins-config-volume\") pod \"007939d3-624d-4e3f-b50a-7644361df5ae\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " Oct 30 03:44:47.196467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196265 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-kube-api-access-qtgch\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-kube-api-access-qtgch\") pod \"007939d3-624d-4e3f-b50a-7644361df5ae\" (UID: \"007939d3-624d-4e3f-b50a-7644361df5ae\") " Oct 30 03:44:47.196662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196431 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-routing-overlay-volume" pod "007939d3-624d-4e3f-b50a-7644361df5ae" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:47.196720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196658 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-plugins-config-volume" pod "007939d3-624d-4e3f-b50a-7644361df5ae" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:47.196845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196826 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:47.196898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.196852 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007939d3-624d-4e3f-b50a-7644361df5ae-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:47.198426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.198406 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials" pod "007939d3-624d-4e3f-b50a-7644361df5ae" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:47.198879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.198835 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-kube-api-access-qtgch" pod "007939d3-624d-4e3f-b50a-7644361df5ae" (UID: "007939d3-624d-4e3f-b50a-7644361df5ae"). InnerVolumeSpecName "kube-api-access-qtgch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:47.235586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.235532 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:47.315810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.298441 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtgch\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-kube-api-access-qtgch\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:47.316087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.316062 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/007939d3-624d-4e3f-b50a-7644361df5ae-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:47.564398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.564360 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:47.620152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.620103 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:47.620358 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:47.620251 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:47.620358 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:47.620280 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj: secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:47.620358 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:47.620350 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials podName:80b700fc-d8d4-45ef-91c6-a189874a0d1f nodeName:}" failed. No retries permitted until 2026-10-30 03:44:55.620335457 +0000 UTC m=+2519.622688636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:44:47.661194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.661166 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:47.785569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.785393 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:47.823731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.823643 4442 generic.go:356] "Generic (PLEG): container finished" podID="007939d3-624d-4e3f-b50a-7644361df5ae" containerID="9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951" exitCode=0 Oct 30 03:44:47.824338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.824251 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" Oct 30 03:44:47.825030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.824274 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" event={"ID":"007939d3-624d-4e3f-b50a-7644361df5ae","Type":"ContainerDied","Data":"9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951"} Oct 30 03:44:47.825505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.825483 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9" event={"ID":"007939d3-624d-4e3f-b50a-7644361df5ae","Type":"ContainerDied","Data":"705056f6ea9b1efc75b5b61154ed90f5128aa4bacbdba7525131614924756528"} Oct 30 03:44:47.825740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.825646 4442 scope.go:122] "RemoveContainer" containerID="9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951" Oct 30 03:44:47.845565 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.845511 4442 scope.go:122] "RemoveContainer" containerID="9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951" Oct 30 03:44:47.845963 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:47.845934 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951\": container with ID starting with 9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951 not found: ID does not exist" containerID="9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951" Oct 30 03:44:47.846054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.845975 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951"} err="failed to get container status \"9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951\": rpc error: code = NotFound desc = could not find container \"9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951\": container with ID starting with 9bd64cae34ea37077b956a4cdf22c764d14458f3f755ce4d5ceaeea5ba8c4951 not found: ID does not exist" Oct 30 03:44:47.912778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.908857 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9"] Oct 30 03:44:47.937806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.934232 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8jqdf9"] Oct 30 03:44:47.986402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:47.986367 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:48.114207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:48.114127 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007939d3-624d-4e3f-b50a-7644361df5ae" path="/var/lib/kubelet/pods/007939d3-624d-4e3f-b50a-7644361df5ae/volumes" Oct 30 03:44:48.663618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:48.663579 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:44:49.108081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.108049 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:49.318247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.318214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:49.420562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.420481 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:49.563860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.563827 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg"] Oct 30 03:44:49.564447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.564432 4442 cpu_manager.go:414] "containerMap: removing container" podUID="007939d3-624d-4e3f-b50a-7644361df5ae" containerName="payload-processing" Oct 30 03:44:49.564509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.564448 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007939d3-624d-4e3f-b50a-7644361df5ae" containerName="payload-processing" Oct 30 03:44:49.564559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.564550 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="007939d3-624d-4e3f-b50a-7644361df5ae" containerName="payload-processing" Oct 30 03:44:49.569385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.569368 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.585375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.585351 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"agc-cred-b25914ccbddc\"" Oct 30 03:44:49.589286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.589264 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg"] Oct 30 03:44:49.596844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.596823 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-e2e--22d06fb9-dockercfg-2mtd5\"" Oct 30 03:44:49.596991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.596968 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"openshift-service-ca.crt\"" Oct 30 03:44:49.596991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.596932 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"routing-overlay\"" Oct 30 03:44:49.596991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.596969 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-plugins-e2e-extmod-a-689bcf\"" Oct 30 03:44:49.597209 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.596921 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"kube-root-ca.crt\"" Oct 30 03:44:49.665801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.662839 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.665801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.662948 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.665801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.662996 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.665801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.663181 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d962d\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-kube-api-access-d962d\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.764160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.764122 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d962d\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-kube-api-access-d962d\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.764338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.764247 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.764338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.764306 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.764480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.764383 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.765604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.765575 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.767805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.767693 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.769808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.769726 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.792842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.785586 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d962d\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-kube-api-access-d962d\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:49.893046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:49.893007 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:50.310856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:50.310780 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:50.323307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:50.323283 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg"] Oct 30 03:44:50.323695 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:44:50.323670 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaafd82_1aad_4f8e_bc81_51f0e379a122.slice/crio-606c5df953a755d4633f3d2c6bab4c7b95c35f8a30e401640405265d3c232da5 WatchSource:0}: Error finding container 606c5df953a755d4633f3d2c6bab4c7b95c35f8a30e401640405265d3c232da5: Status 404 returned error can't find the container with id 606c5df953a755d4633f3d2c6bab4c7b95c35f8a30e401640405265d3c232da5 Oct 30 03:44:50.820639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:50.820606 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:50.847214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:50.847178 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" event={"ID":"adaafd82-1aad-4f8e-bc81-51f0e379a122","Type":"ContainerStarted","Data":"5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32"} Oct 30 03:44:50.847371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:50.847230 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" event={"ID":"adaafd82-1aad-4f8e-bc81-51f0e379a122","Type":"ContainerStarted","Data":"606c5df953a755d4633f3d2c6bab4c7b95c35f8a30e401640405265d3c232da5"} Oct 30 03:44:50.847371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:50.847308 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:44:52.122942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.122911 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:52.336919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.336889 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:52.461809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.461723 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:52.744542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.744448 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:44:52.780319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.780263 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" podStartSLOduration=3.780245515 podStartE2EDuration="3.780245515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:50.909201587 +0000 UTC m=+2514.911554785" watchObservedRunningTime="2026-10-30 03:44:52.780245515 +0000 UTC m=+2516.782598717" Oct 30 03:44:52.810329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.810296 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj"] Oct 30 03:44:52.810792 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:52.810744 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" podUID="80b700fc-d8d4-45ef-91c6-a189874a0d1f" Oct 30 03:44:52.859969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.859939 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:52.868410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.868390 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:52.935160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.935128 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-kube-api-access-v26tw\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-kube-api-access-v26tw\") pod \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " Oct 30 03:44:52.935348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.935174 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-plugins-config-volume\") pod \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " Oct 30 03:44:52.935348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.935300 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-routing-overlay-volume\") pod \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\" (UID: \"80b700fc-d8d4-45ef-91c6-a189874a0d1f\") " Oct 30 03:44:52.935604 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.935580 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-plugins-config-volume" pod "80b700fc-d8d4-45ef-91c6-a189874a0d1f" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:52.935689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.935661 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-routing-overlay-volume" pod "80b700fc-d8d4-45ef-91c6-a189874a0d1f" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:44:52.936171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.936138 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:52.936171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.936171 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80b700fc-d8d4-45ef-91c6-a189874a0d1f-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:52.937587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:52.937566 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-kube-api-access-v26tw" pod "80b700fc-d8d4-45ef-91c6-a189874a0d1f" (UID: "80b700fc-d8d4-45ef-91c6-a189874a0d1f"). InnerVolumeSpecName "kube-api-access-v26tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:44:53.037044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.037015 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26tw\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-kube-api-access-v26tw\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:53.431338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.431249 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:53.839442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.839410 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v"] Oct 30 03:44:53.844021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.844000 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.857562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.857535 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-81afe3-openshift-default-dockercfg-nstjm\"" Oct 30 03:44:53.868729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.868705 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj" Oct 30 03:44:53.872981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.872956 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v"] Oct 30 03:44:53.883912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.883883 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:53.947483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.947455 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj"] Oct 30 03:44:53.947904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.947878 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622786e-9c38-468e-8da1-6522e412e5b0-istiod-ca-cert\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.947971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.947937 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-socket\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.948028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.948008 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622786e-9c38-468e-8da1-6522e412e5b0-istio-podinfo\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.948072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.948063 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-data\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.948188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.948176 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-certs\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.948273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.948203 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxl7\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-kube-api-access-tgxl7\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.948273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.948222 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-envoy\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.948273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.948241 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-istio-token\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.948439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.948396 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-credential-socket\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:53.959959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:53.959937 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-b9b88vmkj"] Oct 30 03:44:54.037045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.037018 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:54.049652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049626 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622786e-9c38-468e-8da1-6522e412e5b0-istiod-ca-cert\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.049831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049667 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-socket\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.049831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049686 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622786e-9c38-468e-8da1-6522e412e5b0-istio-podinfo\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.049831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049738 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-data\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049849 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-certs\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049902 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxl7\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-kube-api-access-tgxl7\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049939 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-envoy\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.049971 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-istio-token\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.050016 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-credential-socket\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.050094 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80b700fc-d8d4-45ef-91c6-a189874a0d1f-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:44:54.050280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.050139 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-data\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.050169 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-socket\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.050379 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-certs\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.050444 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-credential-socket\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.050845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.050822 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622786e-9c38-468e-8da1-6522e412e5b0-istiod-ca-cert\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.052458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.052438 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-envoy\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.052557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.052541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622786e-9c38-468e-8da1-6522e412e5b0-istio-podinfo\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.132961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.132884 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxl7\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-kube-api-access-tgxl7\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.133690 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.133620 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-istio-token\") pod \"e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.189774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.187137 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:54.189774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.188844 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b700fc-d8d4-45ef-91c6-a189874a0d1f" path="/var/lib/kubelet/pods/80b700fc-d8d4-45ef-91c6-a189874a0d1f/volumes" Oct 30 03:44:54.189774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.189267 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:54.485721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.485652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:54.549643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.549619 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v"] Oct 30 03:44:54.550332 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:44:54.550309 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8622786e_9c38_468e_8da1_6522e412e5b0.slice/crio-fd1e7e43049a742e4a0e1390dd143228656c61035aa527426a025f4371a006d2 WatchSource:0}: Error finding container fd1e7e43049a742e4a0e1390dd143228656c61035aa527426a025f4371a006d2: Status 404 returned error can't find the container with id fd1e7e43049a742e4a0e1390dd143228656c61035aa527426a025f4371a006d2 Oct 30 03:44:54.554165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.554126 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:44:54.554379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.554356 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:44:54.554529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.554510 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:44:54.601362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.601337 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:54.701296 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.701264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:54.864241 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:44:54.864206 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:44:54.878048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.877991 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" event={"ID":"8622786e-9c38-468e-8da1-6522e412e5b0","Type":"ContainerStarted","Data":"b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14"} Oct 30 03:44:54.878245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.878050 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" event={"ID":"8622786e-9c38-468e-8da1-6522e412e5b0","Type":"ContainerStarted","Data":"fd1e7e43049a742e4a0e1390dd143228656c61035aa527426a025f4371a006d2"} Oct 30 03:44:54.922293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:54.922242 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podStartSLOduration=1.922228297 podStartE2EDuration="1.922228297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:44:54.918567851 +0000 UTC m=+2518.920921050" watchObservedRunningTime="2026-10-30 03:44:54.922228297 +0000 UTC m=+2518.924581493" Oct 30 03:44:55.190115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:55.190024 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:55.196157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:55.196136 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:55.686051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:55.686021 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:55.884217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:55.884197 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:55.885227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:55.885207 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:44:57.266058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:57.266024 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:57.395323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:57.395289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:58.907084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:58.907045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:44:59.026673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:44:59.026642 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:00.195307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.195276 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl"] Oct 30 03:45:00.201654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.201628 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj"] Oct 30 03:45:00.201816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.201792 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.206271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.206247 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.212663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.212636 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5"] Oct 30 03:45:00.218620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.218310 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.245623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.239772 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdc5037-fae6-4d1d-9b09-7688cea10526-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.245623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.239855 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cvzb\" (UniqueName: \"kubernetes.io/projected/4c673428-1d3c-480f-a7b6-4a42cf5608ff-kube-api-access-4cvzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.245623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.239881 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqwb\" (UniqueName: \"kubernetes.io/projected/acdc5037-fae6-4d1d-9b09-7688cea10526-kube-api-access-9xqwb\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.245623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.239915 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c673428-1d3c-480f-a7b6-4a42cf5608ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.245623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.240161 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b8t\" (UniqueName: \"kubernetes.io/projected/8360c96e-7617-42b6-96d2-edc8bfb60345-kube-api-access-99b8t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.245623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.240216 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8360c96e-7617-42b6-96d2-edc8bfb60345-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.256789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.256708 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj"] Oct 30 03:45:00.261591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.261564 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl"] Oct 30 03:45:00.274905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.274877 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h"] Oct 30 03:45:00.280316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.280287 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5"] Oct 30 03:45:00.280434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.280337 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h"] Oct 30 03:45:00.280499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.280432 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.286988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.286388 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 03:45:00.341126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341097 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9lwp\" (UniqueName: \"kubernetes.io/projected/f17b2bde-61fe-4fff-993b-0a0efafb7de4-kube-api-access-p9lwp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.341261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341149 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b8t\" (UniqueName: \"kubernetes.io/projected/8360c96e-7617-42b6-96d2-edc8bfb60345-kube-api-access-99b8t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.341261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341182 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8360c96e-7617-42b6-96d2-edc8bfb60345-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.341381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341325 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17b2bde-61fe-4fff-993b-0a0efafb7de4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.341460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341442 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdc5037-fae6-4d1d-9b09-7688cea10526-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.341535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341521 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cvzb\" (UniqueName: \"kubernetes.io/projected/4c673428-1d3c-480f-a7b6-4a42cf5608ff-kube-api-access-4cvzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.341570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341550 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqwb\" (UniqueName: \"kubernetes.io/projected/acdc5037-fae6-4d1d-9b09-7688cea10526-kube-api-access-9xqwb\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.341800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341574 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c673428-1d3c-480f-a7b6-4a42cf5608ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.341881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.341852 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8360c96e-7617-42b6-96d2-edc8bfb60345-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.342133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.342116 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdc5037-fae6-4d1d-9b09-7688cea10526-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.342307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.342291 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c673428-1d3c-480f-a7b6-4a42cf5608ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.358725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.358697 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqwb\" (UniqueName: \"kubernetes.io/projected/acdc5037-fae6-4d1d-9b09-7688cea10526-kube-api-access-9xqwb\") pod \"maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.358843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.358745 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b8t\" (UniqueName: \"kubernetes.io/projected/8360c96e-7617-42b6-96d2-edc8bfb60345-kube-api-access-99b8t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.358944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.358924 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cvzb\" (UniqueName: \"kubernetes.io/projected/4c673428-1d3c-480f-a7b6-4a42cf5608ff-kube-api-access-4cvzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.444240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.444199 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9lwp\" (UniqueName: \"kubernetes.io/projected/f17b2bde-61fe-4fff-993b-0a0efafb7de4-kube-api-access-p9lwp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.450696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.450601 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17b2bde-61fe-4fff-993b-0a0efafb7de4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.452275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.451745 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17b2bde-61fe-4fff-993b-0a0efafb7de4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.471500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.471473 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9lwp\" (UniqueName: \"kubernetes.io/projected/f17b2bde-61fe-4fff-993b-0a0efafb7de4-kube-api-access-p9lwp\") pod \"maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.536987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.536957 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:00.543990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.543970 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:00.591059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.590774 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:00.596568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.596311 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:00.603199 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.602379 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:00.890263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.890239 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl"] Oct 30 03:45:00.890708 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:45:00.890680 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8360c96e_7617_42b6_96d2_edc8bfb60345.slice/crio-41591ddb384c7e3ba7b12890fa8067c8a1e9c5921d0990223ee5ac141cab0658 WatchSource:0}: Error finding container 41591ddb384c7e3ba7b12890fa8067c8a1e9c5921d0990223ee5ac141cab0658: Status 404 returned error can't find the container with id 41591ddb384c7e3ba7b12890fa8067c8a1e9c5921d0990223ee5ac141cab0658 Oct 30 03:45:00.920331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.920293 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" event={"ID":"8360c96e-7617-42b6-96d2-edc8bfb60345","Type":"ContainerStarted","Data":"41591ddb384c7e3ba7b12890fa8067c8a1e9c5921d0990223ee5ac141cab0658"} Oct 30 03:45:00.921569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.921547 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj"] Oct 30 03:45:00.922550 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:45:00.922527 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacdc5037_fae6_4d1d_9b09_7688cea10526.slice/crio-4aba29ce7c4ded052dc7a4ed1d9bfca83fdfbc63fdc2488f6e0de21446b93f6a WatchSource:0}: Error finding container 4aba29ce7c4ded052dc7a4ed1d9bfca83fdfbc63fdc2488f6e0de21446b93f6a: Status 404 returned error can't find the container with id 4aba29ce7c4ded052dc7a4ed1d9bfca83fdfbc63fdc2488f6e0de21446b93f6a Oct 30 03:45:00.942681 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:45:00.942656 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17b2bde_61fe_4fff_993b_0a0efafb7de4.slice/crio-378bf3bdf4f42453b0cbe77c89a76e941aa7839d0e0fc736d1d8b6c9faa3c801 WatchSource:0}: Error finding container 378bf3bdf4f42453b0cbe77c89a76e941aa7839d0e0fc736d1d8b6c9faa3c801: Status 404 returned error can't find the container with id 378bf3bdf4f42453b0cbe77c89a76e941aa7839d0e0fc736d1d8b6c9faa3c801 Oct 30 03:45:00.942861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.942837 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h"] Oct 30 03:45:00.952820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:00.952794 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5"] Oct 30 03:45:00.952942 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:45:00.952916 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c673428_1d3c_480f_a7b6_4a42cf5608ff.slice/crio-8c7301993ce9f33199a77093a18807fe004dd8960542253e2b1987fb02cbb746 WatchSource:0}: Error finding container 8c7301993ce9f33199a77093a18807fe004dd8960542253e2b1987fb02cbb746: Status 404 returned error can't find the container with id 8c7301993ce9f33199a77093a18807fe004dd8960542253e2b1987fb02cbb746 Oct 30 03:45:01.228046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.227965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:01.370928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.370890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:01.479958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.479883 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:01.711902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.711850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:01.814966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.814918 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:01.858366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.857266 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:45:01.954643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.954529 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:01.962869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.962822 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" event={"ID":"4c673428-1d3c-480f-a7b6-4a42cf5608ff","Type":"ContainerStarted","Data":"8c7301993ce9f33199a77093a18807fe004dd8960542253e2b1987fb02cbb746"} Oct 30 03:45:01.965689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.965651 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" event={"ID":"f17b2bde-61fe-4fff-993b-0a0efafb7de4","Type":"ContainerStarted","Data":"378bf3bdf4f42453b0cbe77c89a76e941aa7839d0e0fc736d1d8b6c9faa3c801"} Oct 30 03:45:01.970040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:01.970007 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" event={"ID":"acdc5037-fae6-4d1d-9b09-7688cea10526","Type":"ContainerStarted","Data":"4aba29ce7c4ded052dc7a4ed1d9bfca83fdfbc63fdc2488f6e0de21446b93f6a"} Oct 30 03:45:02.089828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:02.089717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:02.266618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:02.266580 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:02.723278 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:02.723247 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:02.827421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:02.827388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:03.901817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:03.901726 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:04.025872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:04.025830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:04.118222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:04.118180 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:04.254702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:04.254633 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:05.811210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:05.811180 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:06.057579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.057547 4442 generic.go:356] "Generic (PLEG): container finished" podID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerID="e2a47f4dc83f03c30a1ca46da6849ff6449eb4abc203b1b812137ff33894d1fd" exitCode=60 Oct 30 03:45:06.057788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.057635 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" event={"ID":"4c673428-1d3c-480f-a7b6-4a42cf5608ff","Type":"ContainerDied","Data":"e2a47f4dc83f03c30a1ca46da6849ff6449eb4abc203b1b812137ff33894d1fd"} Oct 30 03:45:06.057938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.057918 4442 scope.go:122] "RemoveContainer" containerID="e2a47f4dc83f03c30a1ca46da6849ff6449eb4abc203b1b812137ff33894d1fd" Oct 30 03:45:06.059322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.059293 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" event={"ID":"8360c96e-7617-42b6-96d2-edc8bfb60345","Type":"ContainerStarted","Data":"7952beec78495285a240facb96840dc107b4683047700ad390681d9a5b03c3c0"} Oct 30 03:45:06.061272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.061003 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" event={"ID":"f17b2bde-61fe-4fff-993b-0a0efafb7de4","Type":"ContainerStarted","Data":"a6c2cec31ba8bacb0eba28e316866f39444938d74e1d985ce64d26452430370a"} Oct 30 03:45:06.062796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.062777 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" event={"ID":"acdc5037-fae6-4d1d-9b09-7688cea10526","Type":"ContainerStarted","Data":"fa4bea2fb590bf01490af9a4486a5c85eae55bcee23aae7d4a537f980ddbc7c9"} Oct 30 03:45:06.134387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.134340 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" podStartSLOduration=1.941492351 podStartE2EDuration="6.134326746s" totalImagesPullingTime="4.192834395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:06.132206878 +0000 UTC m=+2530.134560085" watchObservedRunningTime="2026-10-30 03:45:06.134326746 +0000 UTC m=+2530.136679943" Oct 30 03:45:06.168891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.168846 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" podStartSLOduration=2.040392877 podStartE2EDuration="6.168834059s" totalImagesPullingTime="4.128441182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:06.167580365 +0000 UTC m=+2530.169933565" watchObservedRunningTime="2026-10-30 03:45:06.168834059 +0000 UTC m=+2530.171187253" Oct 30 03:45:06.205547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:06.205498 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" podStartSLOduration=1.933076005 podStartE2EDuration="6.205485331s" totalImagesPullingTime="4.272409326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:06.20228958 +0000 UTC m=+2530.204642777" watchObservedRunningTime="2026-10-30 03:45:06.205485331 +0000 UTC m=+2530.207838528" Oct 30 03:45:07.073360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.073315 4442 generic.go:356] "Generic (PLEG): container finished" podID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerID="fa4bea2fb590bf01490af9a4486a5c85eae55bcee23aae7d4a537f980ddbc7c9" exitCode=60 Oct 30 03:45:07.073890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.073576 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" event={"ID":"acdc5037-fae6-4d1d-9b09-7688cea10526","Type":"ContainerDied","Data":"fa4bea2fb590bf01490af9a4486a5c85eae55bcee23aae7d4a537f980ddbc7c9"} Oct 30 03:45:07.074199 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.074172 4442 scope.go:122] "RemoveContainer" containerID="fa4bea2fb590bf01490af9a4486a5c85eae55bcee23aae7d4a537f980ddbc7c9" Oct 30 03:45:07.078228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.078193 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" event={"ID":"4c673428-1d3c-480f-a7b6-4a42cf5608ff","Type":"ContainerStarted","Data":"e14cc5276eb9ffa2d8b3013e4e39dad37c97b143767a56d8949f5c7efbe106fe"} Oct 30 03:45:07.081607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.081577 4442 generic.go:356] "Generic (PLEG): container finished" podID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerID="7952beec78495285a240facb96840dc107b4683047700ad390681d9a5b03c3c0" exitCode=60 Oct 30 03:45:07.081745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.081717 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" event={"ID":"8360c96e-7617-42b6-96d2-edc8bfb60345","Type":"ContainerDied","Data":"7952beec78495285a240facb96840dc107b4683047700ad390681d9a5b03c3c0"} Oct 30 03:45:07.082102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.082083 4442 scope.go:122] "RemoveContainer" containerID="7952beec78495285a240facb96840dc107b4683047700ad390681d9a5b03c3c0" Oct 30 03:45:07.087744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.087712 4442 generic.go:356] "Generic (PLEG): container finished" podID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerID="a6c2cec31ba8bacb0eba28e316866f39444938d74e1d985ce64d26452430370a" exitCode=60 Oct 30 03:45:07.087874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.087778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" event={"ID":"f17b2bde-61fe-4fff-993b-0a0efafb7de4","Type":"ContainerDied","Data":"a6c2cec31ba8bacb0eba28e316866f39444938d74e1d985ce64d26452430370a"} Oct 30 03:45:07.088723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.088703 4442 scope.go:122] "RemoveContainer" containerID="a6c2cec31ba8bacb0eba28e316866f39444938d74e1d985ce64d26452430370a" Oct 30 03:45:07.145818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.145770 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" podStartSLOduration=3.217141529 podStartE2EDuration="7.145737306s" totalImagesPullingTime="3.928595777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:07.144006325 +0000 UTC m=+2531.146359525" watchObservedRunningTime="2026-10-30 03:45:07.145737306 +0000 UTC m=+2531.148090555" Oct 30 03:45:07.417235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:07.417211 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:08.097000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.096970 4442 generic.go:356] "Generic (PLEG): container finished" podID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerID="e14cc5276eb9ffa2d8b3013e4e39dad37c97b143767a56d8949f5c7efbe106fe" exitCode=60 Oct 30 03:45:08.097385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.097054 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" event={"ID":"4c673428-1d3c-480f-a7b6-4a42cf5608ff","Type":"ContainerDied","Data":"e14cc5276eb9ffa2d8b3013e4e39dad37c97b143767a56d8949f5c7efbe106fe"} Oct 30 03:45:08.097385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.097119 4442 scope.go:122] "RemoveContainer" containerID="e2a47f4dc83f03c30a1ca46da6849ff6449eb4abc203b1b812137ff33894d1fd" Oct 30 03:45:08.097385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.097354 4442 scope.go:122] "RemoveContainer" containerID="e14cc5276eb9ffa2d8b3013e4e39dad37c97b143767a56d8949f5c7efbe106fe" Oct 30 03:45:08.097681 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:08.097660 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5_odh-ai-gateway-infra(4c673428-1d3c-480f-a7b6-4a42cf5608ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" Oct 30 03:45:08.101646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.101623 4442 generic.go:356] "Generic (PLEG): container finished" podID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerID="96710483323b2c105b505791ea3c083ff4881bd7dfd1ab5b84d320a980711ab0" exitCode=60 Oct 30 03:45:08.102113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.102096 4442 scope.go:122] "RemoveContainer" containerID="96710483323b2c105b505791ea3c083ff4881bd7dfd1ab5b84d320a980711ab0" Oct 30 03:45:08.105355 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:08.102414 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl_odh-ai-gateway-infra(8360c96e-7617-42b6-96d2-edc8bfb60345)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" Oct 30 03:45:08.107197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.107178 4442 generic.go:356] "Generic (PLEG): container finished" podID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerID="5151b3e51deb69428196bc6b9a9d93fb3a0b260a6247c611154282d4e721d67a" exitCode=60 Oct 30 03:45:08.107702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.107684 4442 scope.go:122] "RemoveContainer" containerID="5151b3e51deb69428196bc6b9a9d93fb3a0b260a6247c611154282d4e721d67a" Oct 30 03:45:08.108083 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:08.108043 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h_odh-ai-gateway-infra(f17b2bde-61fe-4fff-993b-0a0efafb7de4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" Oct 30 03:45:08.109663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.109645 4442 scope.go:122] "RemoveContainer" containerID="da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328" Oct 30 03:45:08.109971 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:08.109944 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj_odh-ai-gateway-infra(acdc5037-fae6-4d1d-9b09-7688cea10526)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" Oct 30 03:45:08.112707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.112682 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" event={"ID":"8360c96e-7617-42b6-96d2-edc8bfb60345","Type":"ContainerDied","Data":"96710483323b2c105b505791ea3c083ff4881bd7dfd1ab5b84d320a980711ab0"} Oct 30 03:45:08.112793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.112744 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" event={"ID":"f17b2bde-61fe-4fff-993b-0a0efafb7de4","Type":"ContainerDied","Data":"5151b3e51deb69428196bc6b9a9d93fb3a0b260a6247c611154282d4e721d67a"} Oct 30 03:45:08.112846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.112791 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" event={"ID":"acdc5037-fae6-4d1d-9b09-7688cea10526","Type":"ContainerStarted","Data":"da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328"} Oct 30 03:45:08.195101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.195076 4442 scope.go:122] "RemoveContainer" containerID="7952beec78495285a240facb96840dc107b4683047700ad390681d9a5b03c3c0" Oct 30 03:45:08.209474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.209452 4442 scope.go:122] "RemoveContainer" containerID="a6c2cec31ba8bacb0eba28e316866f39444938d74e1d985ce64d26452430370a" Oct 30 03:45:08.807943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:08.807899 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:09.117561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:09.117478 4442 scope.go:122] "RemoveContainer" containerID="e14cc5276eb9ffa2d8b3013e4e39dad37c97b143767a56d8949f5c7efbe106fe" Oct 30 03:45:09.118029 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:09.117766 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5_odh-ai-gateway-infra(4c673428-1d3c-480f-a7b6-4a42cf5608ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" Oct 30 03:45:09.119206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:09.119183 4442 scope.go:122] "RemoveContainer" containerID="96710483323b2c105b505791ea3c083ff4881bd7dfd1ab5b84d320a980711ab0" Oct 30 03:45:09.119443 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:09.119417 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl_odh-ai-gateway-infra(8360c96e-7617-42b6-96d2-edc8bfb60345)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" Oct 30 03:45:09.120658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:09.120636 4442 scope.go:122] "RemoveContainer" containerID="5151b3e51deb69428196bc6b9a9d93fb3a0b260a6247c611154282d4e721d67a" Oct 30 03:45:09.120900 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:09.120871 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h_odh-ai-gateway-infra(f17b2bde-61fe-4fff-993b-0a0efafb7de4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" Oct 30 03:45:09.122388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:09.122366 4442 generic.go:356] "Generic (PLEG): container finished" podID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerID="da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328" exitCode=60 Oct 30 03:45:09.122474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:09.122450 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" event={"ID":"acdc5037-fae6-4d1d-9b09-7688cea10526","Type":"ContainerDied","Data":"da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328"} Oct 30 03:45:09.122532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:09.122516 4442 scope.go:122] "RemoveContainer" containerID="fa4bea2fb590bf01490af9a4486a5c85eae55bcee23aae7d4a537f980ddbc7c9" Oct 30 03:45:09.122687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:09.122671 4442 scope.go:122] "RemoveContainer" containerID="da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328" Oct 30 03:45:09.122957 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:09.122935 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj_odh-ai-gateway-infra(acdc5037-fae6-4d1d-9b09-7688cea10526)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" Oct 30 03:45:10.131410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:10.131382 4442 scope.go:122] "RemoveContainer" containerID="da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328" Oct 30 03:45:10.131918 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:10.131704 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj_odh-ai-gateway-infra(acdc5037-fae6-4d1d-9b09-7688cea10526)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" Oct 30 03:45:10.424361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:10.424251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:11.900929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:11.900893 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:13.393738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:13.393707 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:14.803876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:14.803830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:16.339043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:16.339004 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:17.938354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:17.938322 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:19.100522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:19.100489 4442 scope.go:122] "RemoveContainer" containerID="da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328" Oct 30 03:45:19.100967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:19.100650 4442 scope.go:122] "RemoveContainer" containerID="e14cc5276eb9ffa2d8b3013e4e39dad37c97b143767a56d8949f5c7efbe106fe" Oct 30 03:45:19.516187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:19.516159 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:20.137486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.137456 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:20.234822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.234798 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:20.241988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.241964 4442 generic.go:356] "Generic (PLEG): container finished" podID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerID="0b34a9ef3ccc0233fcfa5f8bdf31780020a0011660c7515f406f06797e3db582" exitCode=60 Oct 30 03:45:20.242111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.241991 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" event={"ID":"acdc5037-fae6-4d1d-9b09-7688cea10526","Type":"ContainerDied","Data":"0b34a9ef3ccc0233fcfa5f8bdf31780020a0011660c7515f406f06797e3db582"} Oct 30 03:45:20.242111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.242066 4442 scope.go:122] "RemoveContainer" containerID="da8f98adb0f7e20e2dc132872eea7a7c987f724e8bdf08ba88812e48bfb28328" Oct 30 03:45:20.242693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.242675 4442 scope.go:122] "RemoveContainer" containerID="0b34a9ef3ccc0233fcfa5f8bdf31780020a0011660c7515f406f06797e3db582" Oct 30 03:45:20.243081 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:20.243058 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj_odh-ai-gateway-infra(acdc5037-fae6-4d1d-9b09-7688cea10526)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" Oct 30 03:45:20.244443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.244425 4442 generic.go:356] "Generic (PLEG): container finished" podID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerID="aca6aab2303ea567bcda31acaa1f95bfbd18fe86e7a6417413e0d9c90dded749" exitCode=60 Oct 30 03:45:20.244522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.244473 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" event={"ID":"4c673428-1d3c-480f-a7b6-4a42cf5608ff","Type":"ContainerDied","Data":"aca6aab2303ea567bcda31acaa1f95bfbd18fe86e7a6417413e0d9c90dded749"} Oct 30 03:45:20.244832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.244818 4442 scope.go:122] "RemoveContainer" containerID="aca6aab2303ea567bcda31acaa1f95bfbd18fe86e7a6417413e0d9c90dded749" Oct 30 03:45:20.245078 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:20.245062 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5_odh-ai-gateway-infra(4c673428-1d3c-480f-a7b6-4a42cf5608ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" Oct 30 03:45:20.269611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:20.269580 4442 scope.go:122] "RemoveContainer" containerID="e14cc5276eb9ffa2d8b3013e4e39dad37c97b143767a56d8949f5c7efbe106fe" Oct 30 03:45:21.099958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.099926 4442 scope.go:122] "RemoveContainer" containerID="5151b3e51deb69428196bc6b9a9d93fb3a0b260a6247c611154282d4e721d67a" Oct 30 03:45:21.100151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.100026 4442 scope.go:122] "RemoveContainer" containerID="96710483323b2c105b505791ea3c083ff4881bd7dfd1ab5b84d320a980711ab0" Oct 30 03:45:21.293928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.293895 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj"] Oct 30 03:45:21.334139 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.334105 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5"] Oct 30 03:45:21.461782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.461744 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64"] Oct 30 03:45:21.468353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.468253 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.486844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.478098 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64"] Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492027 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\"" Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492090 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\"" Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492576 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-0e64cda2603e\"" Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492658 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-t5k2w\"" Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492718 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-6ad5fa507890\"" Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492784 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-7e237115c4ea\"" Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492870 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\"" Oct 30 03:45:21.493815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.492929 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\"" Oct 30 03:45:21.500988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.500909 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:21.574563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.574533 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:21.594265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.594231 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.594413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.594291 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.594545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.594520 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzcw\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-kube-api-access-pjzcw\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.594686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.594673 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.615695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.615662 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:21.695688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.695556 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c673428-1d3c-480f-a7b6-4a42cf5608ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c673428-1d3c-480f-a7b6-4a42cf5608ff-maas-api-service-ca\") pod \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " Oct 30 03:45:21.695688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.695625 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c673428-1d3c-480f-a7b6-4a42cf5608ff-kube-api-access-4cvzb\" (UniqueName: \"kubernetes.io/projected/4c673428-1d3c-480f-a7b6-4a42cf5608ff-kube-api-access-4cvzb\") pod \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\" (UID: \"4c673428-1d3c-480f-a7b6-4a42cf5608ff\") " Oct 30 03:45:21.695932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.695742 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdc5037-fae6-4d1d-9b09-7688cea10526-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdc5037-fae6-4d1d-9b09-7688cea10526-maas-api-service-ca\") pod \"acdc5037-fae6-4d1d-9b09-7688cea10526\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " Oct 30 03:45:21.695932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.695824 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdc5037-fae6-4d1d-9b09-7688cea10526-kube-api-access-9xqwb\" (UniqueName: \"kubernetes.io/projected/acdc5037-fae6-4d1d-9b09-7688cea10526-kube-api-access-9xqwb\") pod \"acdc5037-fae6-4d1d-9b09-7688cea10526\" (UID: \"acdc5037-fae6-4d1d-9b09-7688cea10526\") " Oct 30 03:45:21.696035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696010 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzcw\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-kube-api-access-pjzcw\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.696140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696014 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c673428-1d3c-480f-a7b6-4a42cf5608ff-maas-api-service-ca" pod "4c673428-1d3c-480f-a7b6-4a42cf5608ff" (UID: "4c673428-1d3c-480f-a7b6-4a42cf5608ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:21.696212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696046 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdc5037-fae6-4d1d-9b09-7688cea10526-maas-api-service-ca" pod "acdc5037-fae6-4d1d-9b09-7688cea10526" (UID: "acdc5037-fae6-4d1d-9b09-7688cea10526"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:21.696266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696218 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.696316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696277 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.696366 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696327 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.696434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696423 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c673428-1d3c-480f-a7b6-4a42cf5608ff-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:21.696489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.696434 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acdc5037-fae6-4d1d-9b09-7688cea10526-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:21.697145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.697119 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.697258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.697172 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.699576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.699552 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.713902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.713881 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzcw\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-kube-api-access-pjzcw\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5869j9k64\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:21.782280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.782242 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdc5037-fae6-4d1d-9b09-7688cea10526-kube-api-access-9xqwb" pod "acdc5037-fae6-4d1d-9b09-7688cea10526" (UID: "acdc5037-fae6-4d1d-9b09-7688cea10526"). InnerVolumeSpecName "kube-api-access-9xqwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:21.782280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.782272 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c673428-1d3c-480f-a7b6-4a42cf5608ff-kube-api-access-4cvzb" pod "4c673428-1d3c-480f-a7b6-4a42cf5608ff" (UID: "4c673428-1d3c-480f-a7b6-4a42cf5608ff"). InnerVolumeSpecName "kube-api-access-4cvzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:21.798259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.798234 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqwb\" (UniqueName: \"kubernetes.io/projected/acdc5037-fae6-4d1d-9b09-7688cea10526-kube-api-access-9xqwb\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:21.798259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.798259 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cvzb\" (UniqueName: \"kubernetes.io/projected/4c673428-1d3c-480f-a7b6-4a42cf5608ff-kube-api-access-4cvzb\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:21.815857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:21.815837 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:22.128193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.128162 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64"] Oct 30 03:45:22.210647 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:45:22.210619 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ba4317_0ca2_4ce2_9857_98bf07a584cb.slice/crio-af58c19c9af1b3bb957864e520e5d4f2b259669ed085704d74e9b0a171fad007 WatchSource:0}: Error finding container af58c19c9af1b3bb957864e520e5d4f2b259669ed085704d74e9b0a171fad007: Status 404 returned error can't find the container with id af58c19c9af1b3bb957864e520e5d4f2b259669ed085704d74e9b0a171fad007 Oct 30 03:45:22.263319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.263289 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" Oct 30 03:45:22.263441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.263320 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj" event={"ID":"acdc5037-fae6-4d1d-9b09-7688cea10526","Type":"ContainerDied","Data":"4aba29ce7c4ded052dc7a4ed1d9bfca83fdfbc63fdc2488f6e0de21446b93f6a"} Oct 30 03:45:22.263441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.263421 4442 scope.go:122] "RemoveContainer" containerID="0b34a9ef3ccc0233fcfa5f8bdf31780020a0011660c7515f406f06797e3db582" Oct 30 03:45:22.264727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.264704 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" event={"ID":"a2ba4317-0ca2-4ce2-9857-98bf07a584cb","Type":"ContainerStarted","Data":"af58c19c9af1b3bb957864e520e5d4f2b259669ed085704d74e9b0a171fad007"} Oct 30 03:45:22.266791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.266772 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" Oct 30 03:45:22.266791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.266781 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5" event={"ID":"4c673428-1d3c-480f-a7b6-4a42cf5608ff","Type":"ContainerDied","Data":"8c7301993ce9f33199a77093a18807fe004dd8960542253e2b1987fb02cbb746"} Oct 30 03:45:22.269125 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.269106 4442 generic.go:356] "Generic (PLEG): container finished" podID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerID="95ef391a6620b7651d01917e4593906568c3b0da7c5a6e54af2113a81083d49d" exitCode=60 Oct 30 03:45:22.269226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.269180 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" event={"ID":"8360c96e-7617-42b6-96d2-edc8bfb60345","Type":"ContainerDied","Data":"95ef391a6620b7651d01917e4593906568c3b0da7c5a6e54af2113a81083d49d"} Oct 30 03:45:22.269647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.269621 4442 scope.go:122] "RemoveContainer" containerID="95ef391a6620b7651d01917e4593906568c3b0da7c5a6e54af2113a81083d49d" Oct 30 03:45:22.269944 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:22.269923 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl_odh-ai-gateway-infra(8360c96e-7617-42b6-96d2-edc8bfb60345)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" Oct 30 03:45:22.271411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.271392 4442 generic.go:356] "Generic (PLEG): container finished" podID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerID="be0a48ed1f5f7408e9c372ac100084b7e41eeecdf5ed6e6c05ce6673ef336cbe" exitCode=60 Oct 30 03:45:22.271489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.271464 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" event={"ID":"f17b2bde-61fe-4fff-993b-0a0efafb7de4","Type":"ContainerDied","Data":"be0a48ed1f5f7408e9c372ac100084b7e41eeecdf5ed6e6c05ce6673ef336cbe"} Oct 30 03:45:22.271799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.271780 4442 scope.go:122] "RemoveContainer" containerID="be0a48ed1f5f7408e9c372ac100084b7e41eeecdf5ed6e6c05ce6673ef336cbe" Oct 30 03:45:22.271995 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:22.271976 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h_odh-ai-gateway-infra(f17b2bde-61fe-4fff-993b-0a0efafb7de4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" Oct 30 03:45:22.275848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.275814 4442 scope.go:122] "RemoveContainer" containerID="aca6aab2303ea567bcda31acaa1f95bfbd18fe86e7a6417413e0d9c90dded749" Oct 30 03:45:22.286023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.285990 4442 scope.go:122] "RemoveContainer" containerID="96710483323b2c105b505791ea3c083ff4881bd7dfd1ab5b84d320a980711ab0" Oct 30 03:45:22.305095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.305071 4442 scope.go:122] "RemoveContainer" containerID="5151b3e51deb69428196bc6b9a9d93fb3a0b260a6247c611154282d4e721d67a" Oct 30 03:45:22.318362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.318336 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5"] Oct 30 03:45:22.334972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.334949 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-817531-29888865-fdzg5"] Oct 30 03:45:22.365603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.365108 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj"] Oct 30 03:45:22.368833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.368813 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-398c25-29888865-qt2pj"] Oct 30 03:45:22.402289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.402265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:22.637256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637178 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6"] Oct 30 03:45:22.637844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637830 4442 cpu_manager.go:414] "containerMap: removing container" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:22.637902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637846 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:22.637902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637875 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.637902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637882 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.637902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637897 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.637902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637902 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.638052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637919 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.638052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637924 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.638052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637949 4442 cpu_manager.go:414] "containerMap: removing container" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:22.638052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.637953 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:22.638174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.638053 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:22.638174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.638061 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:22.638174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.638071 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.638174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.638077 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:22.643084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.643057 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.659638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.659471 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-e2e--4f8d4144-dockercfg-hx7k2\"" Oct 30 03:45:22.663158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.663110 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"openshift-service-ca.crt\"" Oct 30 03:45:22.663158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.663153 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e10c0f\"" Oct 30 03:45:22.663350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.663283 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"kube-root-ca.crt\"" Oct 30 03:45:22.664070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.663929 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"routing-overlay\"" Oct 30 03:45:22.665104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.665082 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6"] Oct 30 03:45:22.714801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.714746 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpbqq\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-kube-api-access-tpbqq\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.714801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.714797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.714994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.714914 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.715032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.715010 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.760567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.760334 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"agc-cred-c4b692a01ee4\"" Oct 30 03:45:22.819372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.816318 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.819372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.816483 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.819372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.816579 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpbqq\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-kube-api-access-tpbqq\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.819372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.816612 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.819372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.818303 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.819372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.819306 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.828218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.828167 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.847740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.847681 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpbqq\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-kube-api-access-tpbqq\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:22.961324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:22.961236 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:23.275414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.275381 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:23.282502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.282173 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" event={"ID":"a2ba4317-0ca2-4ce2-9857-98bf07a584cb","Type":"ContainerStarted","Data":"c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f"} Oct 30 03:45:23.282502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.282310 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:23.284223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.284196 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6"] Oct 30 03:45:23.284404 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:45:23.284380 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cce333_86a0_44f4_9df8_a2caafc67b8a.slice/crio-d5a3ed2f4afea24120a8b8198f9588f02a0ff5ea38dd6371859c1a296e9d87a1 WatchSource:0}: Error finding container d5a3ed2f4afea24120a8b8198f9588f02a0ff5ea38dd6371859c1a296e9d87a1: Status 404 returned error can't find the container with id d5a3ed2f4afea24120a8b8198f9588f02a0ff5ea38dd6371859c1a296e9d87a1 Oct 30 03:45:23.321151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.321109 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" podStartSLOduration=2.321095122 podStartE2EDuration="2.321095122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:23.318564956 +0000 UTC m=+2547.320918153" watchObservedRunningTime="2026-10-30 03:45:23.321095122 +0000 UTC m=+2547.323448318" Oct 30 03:45:23.417595 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.417551 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h"] Oct 30 03:45:23.451029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.450999 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl"] Oct 30 03:45:23.596558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.596511 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:23.626731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.626706 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17b2bde-61fe-4fff-993b-0a0efafb7de4-kube-api-access-p9lwp\" (UniqueName: \"kubernetes.io/projected/f17b2bde-61fe-4fff-993b-0a0efafb7de4-kube-api-access-p9lwp\") pod \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " Oct 30 03:45:23.626854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.626784 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17b2bde-61fe-4fff-993b-0a0efafb7de4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17b2bde-61fe-4fff-993b-0a0efafb7de4-maas-api-service-ca\") pod \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\" (UID: \"f17b2bde-61fe-4fff-993b-0a0efafb7de4\") " Oct 30 03:45:23.627170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.627140 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17b2bde-61fe-4fff-993b-0a0efafb7de4-maas-api-service-ca" pod "f17b2bde-61fe-4fff-993b-0a0efafb7de4" (UID: "f17b2bde-61fe-4fff-993b-0a0efafb7de4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:23.627386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.627369 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17b2bde-61fe-4fff-993b-0a0efafb7de4-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:23.629030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.629010 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17b2bde-61fe-4fff-993b-0a0efafb7de4-kube-api-access-p9lwp" pod "f17b2bde-61fe-4fff-993b-0a0efafb7de4" (UID: "f17b2bde-61fe-4fff-993b-0a0efafb7de4"). InnerVolumeSpecName "kube-api-access-p9lwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:23.647192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.647162 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:23.730797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.730733 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8360c96e-7617-42b6-96d2-edc8bfb60345-kube-api-access-99b8t\" (UniqueName: \"kubernetes.io/projected/8360c96e-7617-42b6-96d2-edc8bfb60345-kube-api-access-99b8t\") pod \"8360c96e-7617-42b6-96d2-edc8bfb60345\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " Oct 30 03:45:23.731000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.730858 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8360c96e-7617-42b6-96d2-edc8bfb60345-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8360c96e-7617-42b6-96d2-edc8bfb60345-maas-api-service-ca\") pod \"8360c96e-7617-42b6-96d2-edc8bfb60345\" (UID: \"8360c96e-7617-42b6-96d2-edc8bfb60345\") " Oct 30 03:45:23.731904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.731449 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9lwp\" (UniqueName: \"kubernetes.io/projected/f17b2bde-61fe-4fff-993b-0a0efafb7de4-kube-api-access-p9lwp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:23.731904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.731823 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8360c96e-7617-42b6-96d2-edc8bfb60345-maas-api-service-ca" pod "8360c96e-7617-42b6-96d2-edc8bfb60345" (UID: "8360c96e-7617-42b6-96d2-edc8bfb60345"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:23.734102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.734074 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8360c96e-7617-42b6-96d2-edc8bfb60345-kube-api-access-99b8t" pod "8360c96e-7617-42b6-96d2-edc8bfb60345" (UID: "8360c96e-7617-42b6-96d2-edc8bfb60345"). InnerVolumeSpecName "kube-api-access-99b8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:23.832808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.832698 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99b8t\" (UniqueName: \"kubernetes.io/projected/8360c96e-7617-42b6-96d2-edc8bfb60345-kube-api-access-99b8t\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:23.832808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:23.832728 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8360c96e-7617-42b6-96d2-edc8bfb60345-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:24.108246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.108163 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" path="/var/lib/kubelet/pods/4c673428-1d3c-480f-a7b6-4a42cf5608ff/volumes" Oct 30 03:45:24.108605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.108591 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" path="/var/lib/kubelet/pods/acdc5037-fae6-4d1d-9b09-7688cea10526/volumes" Oct 30 03:45:24.294976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.294946 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" Oct 30 03:45:24.295167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.294940 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl" event={"ID":"8360c96e-7617-42b6-96d2-edc8bfb60345","Type":"ContainerDied","Data":"41591ddb384c7e3ba7b12890fa8067c8a1e9c5921d0990223ee5ac141cab0658"} Oct 30 03:45:24.295167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.295072 4442 scope.go:122] "RemoveContainer" containerID="95ef391a6620b7651d01917e4593906568c3b0da7c5a6e54af2113a81083d49d" Oct 30 03:45:24.296398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.296368 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" event={"ID":"f17b2bde-61fe-4fff-993b-0a0efafb7de4","Type":"ContainerDied","Data":"378bf3bdf4f42453b0cbe77c89a76e941aa7839d0e0fc736d1d8b6c9faa3c801"} Oct 30 03:45:24.296398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.296382 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h" Oct 30 03:45:24.298803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.298778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" event={"ID":"a7cce333-86a0-44f4-9df8-a2caafc67b8a","Type":"ContainerStarted","Data":"3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f"} Oct 30 03:45:24.298942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.298821 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" event={"ID":"a7cce333-86a0-44f4-9df8-a2caafc67b8a","Type":"ContainerStarted","Data":"d5a3ed2f4afea24120a8b8198f9588f02a0ff5ea38dd6371859c1a296e9d87a1"} Oct 30 03:45:24.298942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.298922 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:24.306068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.305921 4442 scope.go:122] "RemoveContainer" containerID="be0a48ed1f5f7408e9c372ac100084b7e41eeecdf5ed6e6c05ce6673ef336cbe" Oct 30 03:45:24.342938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.342912 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h"] Oct 30 03:45:24.352349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.352326 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a9c9b6-29888865-t9v9h"] Oct 30 03:45:24.387446 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.387370 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" podStartSLOduration=2.387359138 podStartE2EDuration="2.387359138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:24.384003621 +0000 UTC m=+2548.386356829" watchObservedRunningTime="2026-10-30 03:45:24.387359138 +0000 UTC m=+2548.389712380" Oct 30 03:45:24.421055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.421029 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl"] Oct 30 03:45:24.426177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.426156 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6ad168-29888865-z8gkl"] Oct 30 03:45:24.861721 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:24.861687 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:45:24.909762 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:24.909731 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:26.110106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:26.110073 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" path="/var/lib/kubelet/pods/8360c96e-7617-42b6-96d2-edc8bfb60345/volumes" Oct 30 03:45:26.110946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:26.110622 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" path="/var/lib/kubelet/pods/f17b2bde-61fe-4fff-993b-0a0efafb7de4/volumes" Oct 30 03:45:26.414127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:26.414035 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:26.972780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:26.972418 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:27.113777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:27.101052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:27.392376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:27.392348 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64"] Oct 30 03:45:27.395533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:27.395504 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:27.915880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:27.915849 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6"] Oct 30 03:45:27.982357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:27.982325 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:28.205273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:28.205187 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:29.124220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:29.124188 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:29.595492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:29.595450 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:31.032338 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:31.032289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:32.393534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.393493 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" podUID="a2ba4317-0ca2-4ce2-9857-98bf07a584cb" containerName="payload-processing" containerID="cri-o://c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f" gracePeriod=25 Oct 30 03:45:32.415407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.415382 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:32.663638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.663585 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:32.792925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.792895 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-routing-overlay-volume\") pod \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " Oct 30 03:45:32.793091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.792959 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-provider-credentials\") pod \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " Oct 30 03:45:32.793091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.793058 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-kube-api-access-pjzcw\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-kube-api-access-pjzcw\") pod \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " Oct 30 03:45:32.793217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.793165 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-plugins-config-volume\") pod \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\" (UID: \"a2ba4317-0ca2-4ce2-9857-98bf07a584cb\") " Oct 30 03:45:32.793332 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.793310 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-routing-overlay-volume" pod "a2ba4317-0ca2-4ce2-9857-98bf07a584cb" (UID: "a2ba4317-0ca2-4ce2-9857-98bf07a584cb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:32.793607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.793576 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-plugins-config-volume" pod "a2ba4317-0ca2-4ce2-9857-98bf07a584cb" (UID: "a2ba4317-0ca2-4ce2-9857-98bf07a584cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:32.793928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.793909 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:32.794012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.793930 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:32.795386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.795363 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-provider-credentials" pod "a2ba4317-0ca2-4ce2-9857-98bf07a584cb" (UID: "a2ba4317-0ca2-4ce2-9857-98bf07a584cb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:32.795474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.795396 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-kube-api-access-pjzcw" pod "a2ba4317-0ca2-4ce2-9857-98bf07a584cb" (UID: "a2ba4317-0ca2-4ce2-9857-98bf07a584cb"). InnerVolumeSpecName "kube-api-access-pjzcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:32.895317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.895287 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:32.895317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.895314 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzcw\" (UniqueName: \"kubernetes.io/projected/a2ba4317-0ca2-4ce2-9857-98bf07a584cb-kube-api-access-pjzcw\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:32.916971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:32.916896 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" podUID="a7cce333-86a0-44f4-9df8-a2caafc67b8a" containerName="payload-processing" containerID="cri-o://3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f" gracePeriod=25 Oct 30 03:45:33.171654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.171591 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:33.300473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.300436 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-provider-credentials\") pod \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " Oct 30 03:45:33.300669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.300485 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-kube-api-access-tpbqq\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-kube-api-access-tpbqq\") pod \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " Oct 30 03:45:33.300669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.300596 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-plugins-config-volume\") pod \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " Oct 30 03:45:33.300898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.300882 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-routing-overlay-volume\") pod \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\" (UID: \"a7cce333-86a0-44f4-9df8-a2caafc67b8a\") " Oct 30 03:45:33.301075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.301052 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-plugins-config-volume" pod "a7cce333-86a0-44f4-9df8-a2caafc67b8a" (UID: "a7cce333-86a0-44f4-9df8-a2caafc67b8a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:33.301405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.301379 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-routing-overlay-volume" pod "a7cce333-86a0-44f4-9df8-a2caafc67b8a" (UID: "a7cce333-86a0-44f4-9df8-a2caafc67b8a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:33.302830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.302805 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-kube-api-access-tpbqq" pod "a7cce333-86a0-44f4-9df8-a2caafc67b8a" (UID: "a7cce333-86a0-44f4-9df8-a2caafc67b8a"). InnerVolumeSpecName "kube-api-access-tpbqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:33.302923 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.302834 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-provider-credentials" pod "a7cce333-86a0-44f4-9df8-a2caafc67b8a" (UID: "a7cce333-86a0-44f4-9df8-a2caafc67b8a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:33.366169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.366138 4442 generic.go:356] "Generic (PLEG): container finished" podID="a7cce333-86a0-44f4-9df8-a2caafc67b8a" containerID="3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f" exitCode=0 Oct 30 03:45:33.366279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.366186 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" Oct 30 03:45:33.366279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.366217 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" event={"ID":"a7cce333-86a0-44f4-9df8-a2caafc67b8a","Type":"ContainerDied","Data":"3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f"} Oct 30 03:45:33.366279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.366274 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6" event={"ID":"a7cce333-86a0-44f4-9df8-a2caafc67b8a","Type":"ContainerDied","Data":"d5a3ed2f4afea24120a8b8198f9588f02a0ff5ea38dd6371859c1a296e9d87a1"} Oct 30 03:45:33.366384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.366296 4442 scope.go:122] "RemoveContainer" containerID="3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f" Oct 30 03:45:33.367852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.367831 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2ba4317-0ca2-4ce2-9857-98bf07a584cb" containerID="c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f" exitCode=0 Oct 30 03:45:33.367984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.367872 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" Oct 30 03:45:33.367984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.367911 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" event={"ID":"a2ba4317-0ca2-4ce2-9857-98bf07a584cb","Type":"ContainerDied","Data":"c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f"} Oct 30 03:45:33.367984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.367975 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64" event={"ID":"a2ba4317-0ca2-4ce2-9857-98bf07a584cb","Type":"ContainerDied","Data":"af58c19c9af1b3bb957864e520e5d4f2b259669ed085704d74e9b0a171fad007"} Oct 30 03:45:33.376278 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.376260 4442 scope.go:122] "RemoveContainer" containerID="3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f" Oct 30 03:45:33.376595 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:33.376567 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f\": container with ID starting with 3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f not found: ID does not exist" containerID="3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f" Oct 30 03:45:33.376673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.376601 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f"} err="failed to get container status \"3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f\": rpc error: code = NotFound desc = could not find container \"3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f\": container with ID starting with 3454500c83d217af7d2790576eb762e0eec92a8bc2b8958cb76add9b6437c01f not found: ID does not exist" Oct 30 03:45:33.376673 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.376619 4442 scope.go:122] "RemoveContainer" containerID="c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f" Oct 30 03:45:33.386306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.386289 4442 scope.go:122] "RemoveContainer" containerID="c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f" Oct 30 03:45:33.386888 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:33.386574 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f\": container with ID starting with c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f not found: ID does not exist" containerID="c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f" Oct 30 03:45:33.386888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.386608 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f"} err="failed to get container status \"c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f\": rpc error: code = NotFound desc = could not find container \"c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f\": container with ID starting with c39c4c6e9c129deb0edfedb2ad4acf6528728222d3affb4b336ce69617fa5d9f not found: ID does not exist" Oct 30 03:45:33.402762 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.402727 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:33.403053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.402772 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:33.403053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.402784 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpbqq\" (UniqueName: \"kubernetes.io/projected/a7cce333-86a0-44f4-9df8-a2caafc67b8a-kube-api-access-tpbqq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:33.403053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.402792 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7cce333-86a0-44f4-9df8-a2caafc67b8a-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:33.414517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.414494 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6"] Oct 30 03:45:33.417904 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.417880 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-5f97r25h6"] Oct 30 03:45:33.448177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.448102 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64"] Oct 30 03:45:33.451150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.451124 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5869j9k64"] Oct 30 03:45:33.885900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.885870 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:33.986127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:33.986092 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:34.108486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:34.108455 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ba4317-0ca2-4ce2-9857-98bf07a584cb" path="/var/lib/kubelet/pods/a2ba4317-0ca2-4ce2-9857-98bf07a584cb/volumes" Oct 30 03:45:34.108901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:34.108885 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7cce333-86a0-44f4-9df8-a2caafc67b8a" path="/var/lib/kubelet/pods/a7cce333-86a0-44f4-9df8-a2caafc67b8a/volumes" Oct 30 03:45:35.374176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:35.374137 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:36.848524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:36.848494 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:38.329044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:38.329008 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:39.804415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:39.804314 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:41.057282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.057248 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:41.159290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.159200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:41.469724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.469631 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:41.519510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.519473 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx"] Oct 30 03:45:41.520132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520117 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a7cce333-86a0-44f4-9df8-a2caafc67b8a" containerName="payload-processing" Oct 30 03:45:41.520181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520133 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7cce333-86a0-44f4-9df8-a2caafc67b8a" containerName="payload-processing" Oct 30 03:45:41.520181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520156 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:41.520181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520161 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:41.520181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520174 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.520181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520179 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520192 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520197 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520209 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520213 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520225 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520230 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520239 4442 cpu_manager.go:414] "containerMap: removing container" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520243 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520250 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2ba4317-0ca2-4ce2-9857-98bf07a584cb" containerName="payload-processing" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520255 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ba4317-0ca2-4ce2-9857-98bf07a584cb" containerName="payload-processing" Oct 30 03:45:41.520335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520338 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520346 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520352 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520359 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdc5037-fae6-4d1d-9b09-7688cea10526" containerName="cleanup" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520364 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ba4317-0ca2-4ce2-9857-98bf07a584cb" containerName="payload-processing" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520370 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7cce333-86a0-44f4-9df8-a2caafc67b8a" containerName="payload-processing" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520377 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c673428-1d3c-480f-a7b6-4a42cf5608ff" containerName="cleanup" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520383 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:41.520651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.520391 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:41.529349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.529328 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.554647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.554458 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\"" Oct 30 03:45:41.554647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.554514 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\"" Oct 30 03:45:41.554647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.554548 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-7e237115c4ea\"" Oct 30 03:45:41.554647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.554558 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-t5k2w\"" Oct 30 03:45:41.554647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.554589 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\"" Oct 30 03:45:41.554647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.554624 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\"" Oct 30 03:45:41.559221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.559192 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx"] Oct 30 03:45:41.638387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.638354 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.638387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.638391 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.638601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.638465 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwb65\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-kube-api-access-dwb65\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.638601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.638535 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.754074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.753806 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.754074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.753875 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.754074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.753906 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwb65\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-kube-api-access-dwb65\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.754074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.753961 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.757609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.755441 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.757609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.755922 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.764636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.764605 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.773167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.773142 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwb65\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-kube-api-access-dwb65\") pod \"payload-processing-external-model-e2e-extmod-b-372480-5587qknpx\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:41.845629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:41.845581 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:42.151813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:42.151786 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx"] Oct 30 03:45:42.152893 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:45:42.152866 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26c406d_1765_4424_a2c6_12d41b15cd8c.slice/crio-bf509ac11d9194b7299c9ccf564a51327858bd79f0a8bf09b3705fb90396760d WatchSource:0}: Error finding container bf509ac11d9194b7299c9ccf564a51327858bd79f0a8bf09b3705fb90396760d: Status 404 returned error can't find the container with id bf509ac11d9194b7299c9ccf564a51327858bd79f0a8bf09b3705fb90396760d Oct 30 03:45:42.450421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:42.450323 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" event={"ID":"f26c406d-1765-4424-a2c6-12d41b15cd8c","Type":"ContainerStarted","Data":"f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180"} Oct 30 03:45:42.450421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:42.450388 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:42.450421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:42.450412 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" event={"ID":"f26c406d-1765-4424-a2c6-12d41b15cd8c","Type":"ContainerStarted","Data":"bf509ac11d9194b7299c9ccf564a51327858bd79f0a8bf09b3705fb90396760d"} Oct 30 03:45:42.487188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:42.487135 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" podStartSLOduration=1.4871179909999999 podStartE2EDuration="1.487117991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:42.485199062 +0000 UTC m=+2566.487552258" watchObservedRunningTime="2026-10-30 03:45:42.487117991 +0000 UTC m=+2566.489471188" Oct 30 03:45:42.957115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:42.957077 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:44.407718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:44.407675 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:45.442050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:45.442019 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:45:45.842466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:45.842439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:47.379499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.379452 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:47.524961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.524928 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx"] Oct 30 03:45:47.525607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.525592 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:47.525658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.525608 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17b2bde-61fe-4fff-993b-0a0efafb7de4" containerName="cleanup" Oct 30 03:45:47.525721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.525712 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8360c96e-7617-42b6-96d2-edc8bfb60345" containerName="cleanup" Oct 30 03:45:47.529535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.529521 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.549006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.548977 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx"] Oct 30 03:45:47.555367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.555346 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-398c25-openshift-default-dockercfg-88fjr\"" Oct 30 03:45:47.640591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640505 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-envoy\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.640591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640542 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-data\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.640808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640683 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-certs\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.640808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640724 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.640808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640773 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-token\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.640909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640870 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-podinfo\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.640909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640899 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-credential-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.640979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640953 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60232eb-dc73-4290-a115-df93a7b1c6f8-istiod-ca-cert\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.641014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.640989 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5jc\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-kube-api-access-vg5jc\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742114 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-envoy\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742148 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-data\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742219 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-certs\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742246 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742276 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-token\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742313 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-podinfo\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742333 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-credential-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742365 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60232eb-dc73-4290-a115-df93a7b1c6f8-istiod-ca-cert\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742397 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5jc\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-kube-api-access-vg5jc\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742565 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-data\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742639 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742685 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-certs\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.742864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.742714 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-credential-socket\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.743261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.743237 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60232eb-dc73-4290-a115-df93a7b1c6f8-istiod-ca-cert\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.744990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.744966 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-envoy\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.745244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.745220 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-podinfo\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.759957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.759933 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-token\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.760056 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.760002 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5jc\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-kube-api-access-vg5jc\") pod \"e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:47.845598 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:47.845557 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:48.205779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.182079 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx"] Oct 30 03:45:48.209405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.209371 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:45:48.209542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.209442 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:45:48.209542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.209475 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:45:48.528152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.528117 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" event={"ID":"d60232eb-dc73-4290-a115-df93a7b1c6f8","Type":"ContainerStarted","Data":"4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542"} Oct 30 03:45:48.528596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.528178 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" event={"ID":"d60232eb-dc73-4290-a115-df93a7b1c6f8","Type":"ContainerStarted","Data":"3786203bf6a7235e0162ace87f7743b0cab52411461d426b608cff9f3afcb4f3"} Oct 30 03:45:48.578063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.578009 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podStartSLOduration=1.577995123 podStartE2EDuration="1.577995123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:45:48.576299782 +0000 UTC m=+2572.578652982" watchObservedRunningTime="2026-10-30 03:45:48.577995123 +0000 UTC m=+2572.580348320" Oct 30 03:45:48.846369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.846280 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:48.949355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:48.949325 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:49.409343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:49.409304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:49.847825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:49.847747 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 03:45:49.848327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:49.847843 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 03:45:50.430779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:50.430700 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:50.629783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:50.625154 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:50.846181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:50.846145 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 03:45:50.846349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:50.846195 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 03:45:51.846377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:51.846339 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 03:45:51.846843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:51.846391 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 03:45:52.338230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:52.338184 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:52.449889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:52.449842 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:52.846538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:52.846506 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 03:45:52.846952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:52.846556 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 03:45:53.123422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:53.123336 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:53.128193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:53.128165 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx"] Oct 30 03:45:53.132209 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:53.132180 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:53.199885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:53.199834 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:53.737545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:53.737512 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:53.846308 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:53.846271 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 03:45:53.846479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:53.846325 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 03:45:54.128925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:54.128884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:54.846174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:54.846131 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" start-of-body= Oct 30 03:45:54.846387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:54.846200 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.173:15021/healthz/ready\": dial tcp 10.134.0.173:15021: connect: connection refused" Oct 30 03:45:54.863478 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:54.863410 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:45:55.694682 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:55.694652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:55.850951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:55.850920 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:55.851461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:55.851446 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:55.852261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:55.852241 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 03:45:56.926956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:56.926909 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:45:56.927429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:56.926981 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:45:57.185411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:57.185328 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:57.631823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:57.631725 4442 generic.go:356] "Generic (PLEG): container finished" podID="8622786e-9c38-468e-8da1-6522e412e5b0" containerID="b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14" exitCode=0 Oct 30 03:45:57.631823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:57.631784 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" event={"ID":"8622786e-9c38-468e-8da1-6522e412e5b0","Type":"ContainerDied","Data":"b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14"} Oct 30 03:45:57.632302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:57.632283 4442 scope.go:122] "RemoveContainer" containerID="b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14" Oct 30 03:45:57.634066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:57.634036 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:45:57.634187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:57.634124 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:45:57.634187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:57.634170 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:45:58.128804 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.128769 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" podUID="f26c406d-1765-4424-a2c6-12d41b15cd8c" containerName="payload-processing" containerID="cri-o://f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180" gracePeriod=25 Oct 30 03:45:58.423249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.423223 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:58.477905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.477871 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-plugins-config-volume\") pod \"f26c406d-1765-4424-a2c6-12d41b15cd8c\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " Oct 30 03:45:58.478090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.477984 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-provider-credentials\") pod \"f26c406d-1765-4424-a2c6-12d41b15cd8c\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " Oct 30 03:45:58.478090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.478038 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-kube-api-access-dwb65\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-kube-api-access-dwb65\") pod \"f26c406d-1765-4424-a2c6-12d41b15cd8c\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " Oct 30 03:45:58.478226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.478138 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-routing-overlay-volume\") pod \"f26c406d-1765-4424-a2c6-12d41b15cd8c\" (UID: \"f26c406d-1765-4424-a2c6-12d41b15cd8c\") " Oct 30 03:45:58.478351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.478319 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-plugins-config-volume" pod "f26c406d-1765-4424-a2c6-12d41b15cd8c" (UID: "f26c406d-1765-4424-a2c6-12d41b15cd8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:58.478567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.478536 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-routing-overlay-volume" pod "f26c406d-1765-4424-a2c6-12d41b15cd8c" (UID: "f26c406d-1765-4424-a2c6-12d41b15cd8c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:45:58.478908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.478890 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.478992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.478913 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f26c406d-1765-4424-a2c6-12d41b15cd8c-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.480932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.480911 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-provider-credentials" pod "f26c406d-1765-4424-a2c6-12d41b15cd8c" (UID: "f26c406d-1765-4424-a2c6-12d41b15cd8c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:58.481045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.480936 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-kube-api-access-dwb65" pod "f26c406d-1765-4424-a2c6-12d41b15cd8c" (UID: "f26c406d-1765-4424-a2c6-12d41b15cd8c"). InnerVolumeSpecName "kube-api-access-dwb65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:45:58.602779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.584636 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.602779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.584676 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwb65\" (UniqueName: \"kubernetes.io/projected/f26c406d-1765-4424-a2c6-12d41b15cd8c-kube-api-access-dwb65\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:45:58.640067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.640029 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" event={"ID":"8622786e-9c38-468e-8da1-6522e412e5b0","Type":"ContainerStarted","Data":"149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d"} Oct 30 03:45:58.641448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.641421 4442 generic.go:356] "Generic (PLEG): container finished" podID="f26c406d-1765-4424-a2c6-12d41b15cd8c" containerID="f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180" exitCode=0 Oct 30 03:45:58.641591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.641479 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" Oct 30 03:45:58.641591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.641500 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" event={"ID":"f26c406d-1765-4424-a2c6-12d41b15cd8c","Type":"ContainerDied","Data":"f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180"} Oct 30 03:45:58.641591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.641550 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx" event={"ID":"f26c406d-1765-4424-a2c6-12d41b15cd8c","Type":"ContainerDied","Data":"bf509ac11d9194b7299c9ccf564a51327858bd79f0a8bf09b3705fb90396760d"} Oct 30 03:45:58.641591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.641580 4442 scope.go:122] "RemoveContainer" containerID="f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180" Oct 30 03:45:58.651617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.651599 4442 scope.go:122] "RemoveContainer" containerID="f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180" Oct 30 03:45:58.651914 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:45:58.651895 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180\": container with ID starting with f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180 not found: ID does not exist" containerID="f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180" Oct 30 03:45:58.651967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.651922 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180"} err="failed to get container status \"f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180\": rpc error: code = NotFound desc = could not find container \"f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180\": container with ID starting with f5a20bec3cd99eeb4562078a05009879084998e48a62226fb87f63138cef7180 not found: ID does not exist" Oct 30 03:45:58.728274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.728240 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx"] Oct 30 03:45:58.731984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.731959 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-5587qknpx"] Oct 30 03:45:58.806788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:58.804403 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:45:59.199780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:59.193126 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:45:59.199780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:59.193246 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 03:45:59.199780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:59.193291 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 03:45:59.976418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:45:59.976389 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:00.059229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:00.058324 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:00.117519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:00.117484 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26c406d-1765-4424-a2c6-12d41b15cd8c" path="/var/lib/kubelet/pods/f26c406d-1765-4424-a2c6-12d41b15cd8c/volumes" Oct 30 03:46:00.138066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:00.138032 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:00.194196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:00.194160 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 03:46:00.194471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:00.194443 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 03:46:00.460209 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:00.460177 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:01.195360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:01.195311 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 03:46:01.195573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:01.195387 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 03:46:02.022909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:02.022870 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:02.200780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:02.196169 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 03:46:02.200780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:02.196234 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 03:46:03.203778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:03.202859 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 03:46:03.203778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:03.202923 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 03:46:03.466016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:03.465937 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:04.194247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:04.194212 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:46:04.194560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:04.194539 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:46:04.195268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:04.195251 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 03:46:04.918226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:04.918185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:06.341281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:06.341251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:06.441721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:06.441692 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:07.319729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:07.319695 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:07.420290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:07.420242 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:08.137625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:08.137585 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:08.731666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:08.731630 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:09.672877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:09.672843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:11.248195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:11.248163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:11.777350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:11.777309 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:46:12.831811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:12.831781 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:13.833022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:13.832991 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:14.545628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:14.545552 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:15.016527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:15.016496 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:15.121837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:15.121802 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:15.298900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:15.298808 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:15.602435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:15.602355 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:15.745705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:15.745671 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:15.845872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:15.845836 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:16.042493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:16.042461 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:16.385580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:16.385491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:16.486304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:16.486260 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:17.366503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:17.366475 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:17.460829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:17.460801 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:17.727369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:17.727264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:18.020118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:18.020089 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:19.261899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:19.261864 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:20.751251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:20.751213 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:22.345550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:22.345496 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:23.186384 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:23.186349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:23.287223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:23.287196 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:23.821201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:23.821164 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:23.919530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:23.919496 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:24.045243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:24.045207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:24.861450 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:24.861414 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:46:25.331611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:25.331581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:26.079394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:26.079362 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc"] Oct 30 03:46:26.091372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:26.091337 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:26.176863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:26.176835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:26.276445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:26.276415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:26.376928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:26.376833 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:26.476686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:26.476649 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:27.211254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.211198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:27.354221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.354172 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:27.443402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.443370 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:27.653045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.653017 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx"] Oct 30 03:46:27.654159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.654134 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f26c406d-1765-4424-a2c6-12d41b15cd8c" containerName="payload-processing" Oct 30 03:46:27.654159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.654159 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26c406d-1765-4424-a2c6-12d41b15cd8c" containerName="payload-processing" Oct 30 03:46:27.654407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.654376 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26c406d-1765-4424-a2c6-12d41b15cd8c" containerName="payload-processing" Oct 30 03:46:27.658527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.658507 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.675734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.675711 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx"] Oct 30 03:46:27.682266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.681487 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"agc-cred-1fa0412344ba\"" Oct 30 03:46:27.682266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.681602 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-9d8692\"/\"payload-processing-external-model-e2e--51bfd021-dockercfg-js4kd\"" Oct 30 03:46:27.811178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.810659 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.811178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.810727 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6l2j\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-kube-api-access-g6l2j\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.811178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.810927 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.811178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.810992 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.914710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.914605 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.915122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.915084 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.915300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.915281 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6l2j\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-kube-api-access-g6l2j\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.915578 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.915560 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.916523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.916498 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.916783 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:27.916743 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:27.918285 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:27.918267 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx: secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:27.918479 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:27.918464 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials podName:441c5e9b-411b-4efa-81be-45e5bdb85d1d nodeName:}" failed. No retries permitted until 2026-10-30 03:46:28.418444568 +0000 UTC m=+2612.420797758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" (UID: "441c5e9b-411b-4efa-81be-45e5bdb85d1d") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:27.918592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.917945 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:27.957159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:27.957088 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6l2j\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-kube-api-access-g6l2j\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:28.020057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.020022 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx"] Oct 30 03:46:28.020436 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:28.020323 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" podUID="441c5e9b-411b-4efa-81be-45e5bdb85d1d" Oct 30 03:46:28.035663 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.035637 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:28.116954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.116925 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:28.425777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.425244 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:28.425777 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:28.425448 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:28.425777 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:28.425464 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx: secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:28.425777 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:28.425529 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials podName:441c5e9b-411b-4efa-81be-45e5bdb85d1d nodeName:}" failed. No retries permitted until 2026-10-30 03:46:29.425511652 +0000 UTC m=+2613.427864827 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" (UID: "441c5e9b-411b-4efa-81be-45e5bdb85d1d") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:28.869122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.869093 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:28.877409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.877386 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:28.932825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.932796 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-plugins-config-volume\") pod \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " Oct 30 03:46:28.933010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.932999 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-routing-overlay-volume\") pod \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " Oct 30 03:46:28.933060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.933027 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-kube-api-access-g6l2j\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-kube-api-access-g6l2j\") pod \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " Oct 30 03:46:28.933197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.933171 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-plugins-config-volume" pod "441c5e9b-411b-4efa-81be-45e5bdb85d1d" (UID: "441c5e9b-411b-4efa-81be-45e5bdb85d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:28.933353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.933333 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-routing-overlay-volume" pod "441c5e9b-411b-4efa-81be-45e5bdb85d1d" (UID: "441c5e9b-411b-4efa-81be-45e5bdb85d1d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:28.933572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.933558 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:28.933621 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.933576 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c5e9b-411b-4efa-81be-45e5bdb85d1d-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:28.935405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:28.935379 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-kube-api-access-g6l2j" pod "441c5e9b-411b-4efa-81be-45e5bdb85d1d" (UID: "441c5e9b-411b-4efa-81be-45e5bdb85d1d"). InnerVolumeSpecName "kube-api-access-g6l2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:29.035151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.035121 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6l2j\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-kube-api-access-g6l2j\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:29.102896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.102847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:29.440658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.440628 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx\" (UID: \"441c5e9b-411b-4efa-81be-45e5bdb85d1d\") " pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:29.441085 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:29.440802 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-9d8692/agc-cred-1fa0412344ba: secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:29.441085 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:29.440820 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx: secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:29.441085 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:29.440879 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials podName:441c5e9b-411b-4efa-81be-45e5bdb85d1d nodeName:}" failed. No retries permitted until 2026-10-30 03:46:31.440865828 +0000 UTC m=+2615.443219019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" (UID: "441c5e9b-411b-4efa-81be-45e5bdb85d1d") : secret "agc-cred-1fa0412344ba" not found Oct 30 03:46:29.538284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.538251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:29.636179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.636145 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg"] Oct 30 03:46:29.660304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.660269 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:29.760708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.760508 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:29.876078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.876049 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx" Oct 30 03:46:29.949930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.949890 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx"] Oct 30 03:46:29.955504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:29.955476 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-59f8kxmmx"] Oct 30 03:46:30.080830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.080003 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/441c5e9b-411b-4efa-81be-45e5bdb85d1d-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:30.133349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.133310 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="441c5e9b-411b-4efa-81be-45e5bdb85d1d" path="/var/lib/kubelet/pods/441c5e9b-411b-4efa-81be-45e5bdb85d1d/volumes" Oct 30 03:46:30.185432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.185368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:30.528671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.528639 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:30.631926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.631873 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:30.729523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.729495 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:30.829339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.829262 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:30.983925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:30.983878 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:31.080597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.080493 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" podUID="001a8986-c6b5-4613-a492-b5ad66989952" containerName="payload-processing" containerID="cri-o://17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1" gracePeriod=25 Oct 30 03:46:31.123178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.123146 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:31.223715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.223683 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:31.357827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.357800 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:46:31.418220 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.418181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:31.434206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.434177 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-kube-api-access-ww7df\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-kube-api-access-ww7df\") pod \"001a8986-c6b5-4613-a492-b5ad66989952\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " Oct 30 03:46:31.434472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.434452 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-plugins-config-volume\") pod \"001a8986-c6b5-4613-a492-b5ad66989952\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " Oct 30 03:46:31.434512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.434496 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-routing-overlay-volume\") pod \"001a8986-c6b5-4613-a492-b5ad66989952\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " Oct 30 03:46:31.434545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.434529 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-provider-credentials\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-provider-credentials\") pod \"001a8986-c6b5-4613-a492-b5ad66989952\" (UID: \"001a8986-c6b5-4613-a492-b5ad66989952\") " Oct 30 03:46:31.435002 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.434966 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-routing-overlay-volume" pod "001a8986-c6b5-4613-a492-b5ad66989952" (UID: "001a8986-c6b5-4613-a492-b5ad66989952"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:31.435132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.435086 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-plugins-config-volume" pod "001a8986-c6b5-4613-a492-b5ad66989952" (UID: "001a8986-c6b5-4613-a492-b5ad66989952"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:31.436714 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.436692 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-kube-api-access-ww7df" pod "001a8986-c6b5-4613-a492-b5ad66989952" (UID: "001a8986-c6b5-4613-a492-b5ad66989952"). InnerVolumeSpecName "kube-api-access-ww7df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:31.436926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.436909 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-provider-credentials" pod "001a8986-c6b5-4613-a492-b5ad66989952" (UID: "001a8986-c6b5-4613-a492-b5ad66989952"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:31.536653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.536617 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:31.536653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.536650 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/001a8986-c6b5-4613-a492-b5ad66989952-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:31.536653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.536660 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:31.537147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.536668 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7df\" (UniqueName: \"kubernetes.io/projected/001a8986-c6b5-4613-a492-b5ad66989952-kube-api-access-ww7df\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:31.934020 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.933981 4442 generic.go:356] "Generic (PLEG): container finished" podID="001a8986-c6b5-4613-a492-b5ad66989952" containerID="17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1" exitCode=0 Oct 30 03:46:31.934201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.934025 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" Oct 30 03:46:31.934201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.934053 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" event={"ID":"001a8986-c6b5-4613-a492-b5ad66989952","Type":"ContainerDied","Data":"17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1"} Oct 30 03:46:31.934201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.934125 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc" event={"ID":"001a8986-c6b5-4613-a492-b5ad66989952","Type":"ContainerDied","Data":"3d9a69c29114b08086bd73d8914fb83f81b25b2fc7a2f43d85c8c186a8ba4cbd"} Oct 30 03:46:31.934201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.934154 4442 scope.go:122] "RemoveContainer" containerID="17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1" Oct 30 03:46:31.944964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.944938 4442 scope.go:122] "RemoveContainer" containerID="17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1" Oct 30 03:46:31.945247 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:31.945229 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1\": container with ID starting with 17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1 not found: ID does not exist" containerID="17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1" Oct 30 03:46:31.945303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.945257 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1"} err="failed to get container status \"17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1\": rpc error: code = NotFound desc = could not find container \"17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1\": container with ID starting with 17cfec74530b9bd06dcc315c142b7eb19d3a69356301feeb870cace82f68c9b1 not found: ID does not exist" Oct 30 03:46:31.984717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.984687 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc"] Oct 30 03:46:31.993882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:31.993860 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-9d8692/payload-processing-external-model-e2e-extmod-a-9d8692-8494djhwc"] Oct 30 03:46:32.082961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:32.082920 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:32.111826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:32.111795 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001a8986-c6b5-4613-a492-b5ad66989952" path="/var/lib/kubelet/pods/001a8986-c6b5-4613-a492-b5ad66989952/volumes" Oct 30 03:46:32.186086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:32.185988 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:32.526777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:32.525222 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:33.261244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.261215 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:33.362559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.362524 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:33.542461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.542349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:33.811943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.811848 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst"] Oct 30 03:46:33.812976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.812951 4442 cpu_manager.go:414] "containerMap: removing container" podUID="001a8986-c6b5-4613-a492-b5ad66989952" containerName="payload-processing" Oct 30 03:46:33.812976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.812976 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001a8986-c6b5-4613-a492-b5ad66989952" containerName="payload-processing" Oct 30 03:46:33.813187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.813116 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="001a8986-c6b5-4613-a492-b5ad66989952" containerName="payload-processing" Oct 30 03:46:33.818779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.818732 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:33.834375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.834350 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst"] Oct 30 03:46:33.841075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.840677 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"openshift-service-ca.crt\"" Oct 30 03:46:33.841075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.840819 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"payload-processing-external-model-e2e--d3502631-dockercfg-nlc98\"" Oct 30 03:46:33.841075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.840892 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"payload-processing-external-model-plugins-e2e-extmod-a-441461\"" Oct 30 03:46:33.841075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.840948 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"agc-cred-f55f38b5809a\"" Oct 30 03:46:33.841075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.841046 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"routing-overlay\"" Oct 30 03:46:33.846121 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.846099 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"kube-root-ca.crt\"" Oct 30 03:46:33.979137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.979104 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:33.979137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.979137 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:33.979406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.979383 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz74t\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-kube-api-access-gz74t\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:33.979454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:33.979442 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.080372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.080293 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz74t\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-kube-api-access-gz74t\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.080372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.080351 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.080592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.080456 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.080592 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.080480 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.081176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.081151 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.081289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.081260 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.083264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.083247 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.097875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.097846 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz74t\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-kube-api-access-gz74t\") pod \"payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.141781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.136011 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.325777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.325681 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:34.425920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.425835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:34.525651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.525621 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:34.550145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.550119 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst"] Oct 30 03:46:34.551379 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:46:34.551356 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6e31a0_e5e3_4ded_9b02_a8c2a5df37c5.slice/crio-8e111344224502cb1509460d0854483d0cb87af40b7d7b1dbadbb14517f5b37f WatchSource:0}: Error finding container 8e111344224502cb1509460d0854483d0cb87af40b7d7b1dbadbb14517f5b37f: Status 404 returned error can't find the container with id 8e111344224502cb1509460d0854483d0cb87af40b7d7b1dbadbb14517f5b37f Oct 30 03:46:34.554192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.554167 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf"] Oct 30 03:46:34.559873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.559853 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.574222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.573174 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-441461\"/\"agc-cred-89629b3d9fc3\"" Oct 30 03:46:34.575943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.575919 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf"] Oct 30 03:46:34.617531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.617498 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.617701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.617537 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.617701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.617669 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.617863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.617797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-kube-api-access-9wjjm\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.637484 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.637450 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" podUID="adaafd82-1aad-4f8e-bc81-51f0e379a122" containerName="payload-processing" containerID="cri-o://5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32" gracePeriod=25 Oct 30 03:46:34.718991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.718910 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-kube-api-access-9wjjm\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.719153 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.719134 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.719228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.719164 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.719293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.719237 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.719913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.719886 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.720052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.720030 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.722013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.721992 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.736392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.736371 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-kube-api-access-9wjjm\") pod \"payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.877819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.877794 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:34.877951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.877908 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:46:34.926040 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.926007 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-provider-credentials\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-provider-credentials\") pod \"adaafd82-1aad-4f8e-bc81-51f0e379a122\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " Oct 30 03:46:34.926197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.926070 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-routing-overlay-volume\") pod \"adaafd82-1aad-4f8e-bc81-51f0e379a122\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " Oct 30 03:46:34.926197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.926100 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-kube-api-access-d962d\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-kube-api-access-d962d\") pod \"adaafd82-1aad-4f8e-bc81-51f0e379a122\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " Oct 30 03:46:34.926286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.926249 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-plugins-config-volume\") pod \"adaafd82-1aad-4f8e-bc81-51f0e379a122\" (UID: \"adaafd82-1aad-4f8e-bc81-51f0e379a122\") " Oct 30 03:46:34.927236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.927203 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-plugins-config-volume" pod "adaafd82-1aad-4f8e-bc81-51f0e379a122" (UID: "adaafd82-1aad-4f8e-bc81-51f0e379a122"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:34.927807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.927781 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-routing-overlay-volume" pod "adaafd82-1aad-4f8e-bc81-51f0e379a122" (UID: "adaafd82-1aad-4f8e-bc81-51f0e379a122"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:34.930349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.930323 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-kube-api-access-d962d" pod "adaafd82-1aad-4f8e-bc81-51f0e379a122" (UID: "adaafd82-1aad-4f8e-bc81-51f0e379a122"). InnerVolumeSpecName "kube-api-access-d962d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:34.945472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.942614 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-provider-credentials" pod "adaafd82-1aad-4f8e-bc81-51f0e379a122" (UID: "adaafd82-1aad-4f8e-bc81-51f0e379a122"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:34.963246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.963214 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" event={"ID":"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5","Type":"ContainerStarted","Data":"7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e"} Oct 30 03:46:34.963520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.963486 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:34.963651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.963536 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" event={"ID":"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5","Type":"ContainerStarted","Data":"8e111344224502cb1509460d0854483d0cb87af40b7d7b1dbadbb14517f5b37f"} Oct 30 03:46:34.965072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.965045 4442 generic.go:356] "Generic (PLEG): container finished" podID="adaafd82-1aad-4f8e-bc81-51f0e379a122" containerID="5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32" exitCode=0 Oct 30 03:46:34.965186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.965092 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" Oct 30 03:46:34.965186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.965116 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" event={"ID":"adaafd82-1aad-4f8e-bc81-51f0e379a122","Type":"ContainerDied","Data":"5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32"} Oct 30 03:46:34.965186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.965155 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg" event={"ID":"adaafd82-1aad-4f8e-bc81-51f0e379a122","Type":"ContainerDied","Data":"606c5df953a755d4633f3d2c6bab4c7b95c35f8a30e401640405265d3c232da5"} Oct 30 03:46:34.965186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.965179 4442 scope.go:122] "RemoveContainer" containerID="5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32" Oct 30 03:46:34.978786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.976670 4442 scope.go:122] "RemoveContainer" containerID="5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32" Oct 30 03:46:34.978786 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:34.977196 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32\": container with ID starting with 5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32 not found: ID does not exist" containerID="5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32" Oct 30 03:46:34.978786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:34.977222 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32"} err="failed to get container status \"5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32\": rpc error: code = NotFound desc = could not find container \"5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32\": container with ID starting with 5c8d6bcb62ef0d371a2adb4d1c8ffa54781b08ee335ec59216683e157efe1b32 not found: ID does not exist" Oct 30 03:46:35.003716 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.003661 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" podStartSLOduration=2.003645115 podStartE2EDuration="2.003645115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:35.000704334 +0000 UTC m=+2619.003057566" watchObservedRunningTime="2026-10-30 03:46:35.003645115 +0000 UTC m=+2619.005998314" Oct 30 03:46:35.028057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.028022 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:35.028057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.028054 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:35.028250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.028071 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/adaafd82-1aad-4f8e-bc81-51f0e379a122-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:35.028250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.028085 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d962d\" (UniqueName: \"kubernetes.io/projected/adaafd82-1aad-4f8e-bc81-51f0e379a122-kube-api-access-d962d\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:35.035674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.035651 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg"] Oct 30 03:46:35.039781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.039739 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-86f567chg"] Oct 30 03:46:35.204980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.204955 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf"] Oct 30 03:46:35.205995 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:46:35.205969 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ec0153_3f7d_46ed_8602_4ca47d5b327d.slice/crio-2636dda0c90bab9e2ab80263f7ac1ba3ea97c7bab1ff46082fbfecd087a91f53 WatchSource:0}: Error finding container 2636dda0c90bab9e2ab80263f7ac1ba3ea97c7bab1ff46082fbfecd087a91f53: Status 404 returned error can't find the container with id 2636dda0c90bab9e2ab80263f7ac1ba3ea97c7bab1ff46082fbfecd087a91f53 Oct 30 03:46:35.855348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.855313 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:35.997023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.996978 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" event={"ID":"62ec0153-3f7d-46ed-8602-4ca47d5b327d","Type":"ContainerStarted","Data":"d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004"} Oct 30 03:46:35.997207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.997052 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" event={"ID":"62ec0153-3f7d-46ed-8602-4ca47d5b327d","Type":"ContainerStarted","Data":"2636dda0c90bab9e2ab80263f7ac1ba3ea97c7bab1ff46082fbfecd087a91f53"} Oct 30 03:46:35.998090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:35.998065 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:36.053607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:36.053540 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" podStartSLOduration=2.053521057 podStartE2EDuration="2.053521057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:36.047940153 +0000 UTC m=+2620.050293350" watchObservedRunningTime="2026-10-30 03:46:36.053521057 +0000 UTC m=+2620.055874505" Oct 30 03:46:36.109372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:36.109293 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adaafd82-1aad-4f8e-bc81-51f0e379a122" path="/var/lib/kubelet/pods/adaafd82-1aad-4f8e-bc81-51f0e379a122/volumes" Oct 30 03:46:36.273302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:36.273275 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:37.206280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:37.206249 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:37.414797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:37.414765 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:38.846479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:38.846441 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:40.330141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:40.330107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:40.483907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:40.483879 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:40.588827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:40.584822 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:41.489431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:41.489399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:41.589571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:41.589545 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:42.054111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:42.054078 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:42.674038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:42.674001 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:42.839147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:42.839105 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:43.047103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.047064 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs"] Oct 30 03:46:43.047832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.047812 4442 cpu_manager.go:414] "containerMap: removing container" podUID="adaafd82-1aad-4f8e-bc81-51f0e379a122" containerName="payload-processing" Oct 30 03:46:43.047832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.047832 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adaafd82-1aad-4f8e-bc81-51f0e379a122" containerName="payload-processing" Oct 30 03:46:43.048049 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.048036 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="adaafd82-1aad-4f8e-bc81-51f0e379a122" containerName="payload-processing" Oct 30 03:46:43.052856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.052833 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.070846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.070815 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs"] Oct 30 03:46:43.077738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.077714 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f0eb942fa2d9\"" Oct 30 03:46:43.077887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.077792 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 03:46:43.077961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.077912 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 03:46:43.078025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.077987 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 03:46:43.078083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.078056 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7a0ff47fb860\"" Oct 30 03:46:43.078951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.078914 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 03:46:43.079083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.078984 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pnpl6\"" Oct 30 03:46:43.154206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.154174 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-plugins-config-volume\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.154206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.154205 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj2fn\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-kube-api-access-lj2fn\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.154486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.154463 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-provider-credentials\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.154576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.154557 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-routing-overlay-volume\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.256054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.256023 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-routing-overlay-volume\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.256201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.256131 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-plugins-config-volume\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.256201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.256155 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj2fn\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-kube-api-access-lj2fn\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.256300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.256279 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-provider-credentials\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.256683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.256663 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-routing-overlay-volume\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.256940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.256920 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-plugins-config-volume\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.259106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.259082 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-provider-credentials\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.274487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.274456 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj2fn\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-kube-api-access-lj2fn\") pod \"payload-processing-external-model-d5bf6f9f-dmvrs\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.367367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.367297 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:43.485009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.484977 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:46:43.797602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.797573 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs"] Oct 30 03:46:43.801107 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:46:43.801075 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6452d4_d4b3_4e11_ba0e_09929fec6bb3.slice/crio-2a207604cadf1107214f93a83a569f2c157455023cbbd05adcd7f10dccadd280 WatchSource:0}: Error finding container 2a207604cadf1107214f93a83a569f2c157455023cbbd05adcd7f10dccadd280: Status 404 returned error can't find the container with id 2a207604cadf1107214f93a83a569f2c157455023cbbd05adcd7f10dccadd280 Oct 30 03:46:43.852061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.852034 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:43.968184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:43.968154 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:44.068051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:44.067969 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:44.071264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:44.071226 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" event={"ID":"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3","Type":"ContainerStarted","Data":"4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd"} Oct 30 03:46:44.071429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:44.071274 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" event={"ID":"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3","Type":"ContainerStarted","Data":"2a207604cadf1107214f93a83a569f2c157455023cbbd05adcd7f10dccadd280"} Oct 30 03:46:44.071429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:44.071338 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:44.110797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:44.110735 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" podStartSLOduration=1.110721854 podStartE2EDuration="1.110721854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:44.107335308 +0000 UTC m=+2628.109688525" watchObservedRunningTime="2026-10-30 03:46:44.110721854 +0000 UTC m=+2628.113075051" Oct 30 03:46:45.206112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.206072 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs"] Oct 30 03:46:45.285913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.285840 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7"] Oct 30 03:46:45.294790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.294252 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.319998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.319735 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-870a31c4296d\"" Oct 30 03:46:45.346166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.343052 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7"] Oct 30 03:46:45.398275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.397781 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jm6\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-kube-api-access-c6jm6\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.398275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.397830 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-plugins-config-volume\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.398275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.398117 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-provider-credentials\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.398275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.398204 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-routing-overlay-volume\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.501212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.501162 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-provider-credentials\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.501409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.501261 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-routing-overlay-volume\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.501409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.501322 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jm6\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-kube-api-access-c6jm6\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.501409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.501359 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-plugins-config-volume\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.502499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.502464 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-routing-overlay-volume\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.502630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.502552 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-plugins-config-volume\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.505813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.505742 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-provider-credentials\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.521520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.521472 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jm6\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-kube-api-access-c6jm6\") pod \"payload-processing-external-model-67d95c9c8-n62m7\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.651138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.651106 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:45.736731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.719051 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:45.965118 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:46:45.965075 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e57a0c_b697_41ec_b42a_c35f625668ba.slice/crio-9bdc4bac5f00b56f2b9bd24f5b8ca20a0f677cbc94f287785983ccb4f8bfe870 WatchSource:0}: Error finding container 9bdc4bac5f00b56f2b9bd24f5b8ca20a0f677cbc94f287785983ccb4f8bfe870: Status 404 returned error can't find the container with id 9bdc4bac5f00b56f2b9bd24f5b8ca20a0f677cbc94f287785983ccb4f8bfe870 Oct 30 03:46:45.965537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:45.965511 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7"] Oct 30 03:46:46.018420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:46.016078 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:46.092606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:46.092309 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" event={"ID":"03e57a0c-b697-41ec-b42a-c35f625668ba","Type":"ContainerStarted","Data":"707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1"} Oct 30 03:46:46.092606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:46.092372 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" event={"ID":"03e57a0c-b697-41ec-b42a-c35f625668ba","Type":"ContainerStarted","Data":"9bdc4bac5f00b56f2b9bd24f5b8ca20a0f677cbc94f287785983ccb4f8bfe870"} Oct 30 03:46:46.092925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:46.092618 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:46.172293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:46.172237 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" podStartSLOduration=1.172222452 podStartE2EDuration="1.172222452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:46.132106223 +0000 UTC m=+2630.134459421" watchObservedRunningTime="2026-10-30 03:46:46.172222452 +0000 UTC m=+2630.174575648" Oct 30 03:46:47.341862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:47.341825 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:47.935919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:47.935890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:48.030343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:48.030317 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:46:48.096848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:48.096814 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst"] Oct 30 03:46:48.894353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:48.894317 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:49.520718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:49.520683 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:50.207797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:50.207214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:50.305309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:50.303875 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:50.557182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:50.557152 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:50.593400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:50.593369 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7"] Oct 30 03:46:50.640328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:50.640292 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:51.092237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.092196 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" podUID="7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" containerName="payload-processing" containerID="cri-o://4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd" gracePeriod=25 Oct 30 03:46:51.361233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.361174 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:51.409875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.409834 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-provider-credentials\") pod \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " Oct 30 03:46:51.410113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.409942 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-kube-api-access-lj2fn\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-kube-api-access-lj2fn\") pod \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " Oct 30 03:46:51.410190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.410115 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-plugins-config-volume\") pod \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " Oct 30 03:46:51.410190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.410150 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-routing-overlay-volume\") pod \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\" (UID: \"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3\") " Oct 30 03:46:51.411319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.411283 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-plugins-config-volume" pod "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" (UID: "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:51.411457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.411429 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-routing-overlay-volume" pod "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" (UID: "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:51.413394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.413366 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-provider-credentials" pod "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" (UID: "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:51.413500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.413474 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-kube-api-access-lj2fn" pod "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" (UID: "7a6452d4-d4b3-4e11-ba0e-09929fec6bb3"). InnerVolumeSpecName "kube-api-access-lj2fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:51.513119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.513032 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj2fn\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-kube-api-access-lj2fn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:51.513119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.513071 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:51.513119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.513086 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:51.513119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:51.513099 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:52.137335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.137303 4442 generic.go:356] "Generic (PLEG): container finished" podID="7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" containerID="4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd" exitCode=0 Oct 30 03:46:52.137497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.137359 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" Oct 30 03:46:52.137497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.137391 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" event={"ID":"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3","Type":"ContainerDied","Data":"4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd"} Oct 30 03:46:52.137497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.137446 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs" event={"ID":"7a6452d4-d4b3-4e11-ba0e-09929fec6bb3","Type":"ContainerDied","Data":"2a207604cadf1107214f93a83a569f2c157455023cbbd05adcd7f10dccadd280"} Oct 30 03:46:52.137497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.137471 4442 scope.go:122] "RemoveContainer" containerID="4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd" Oct 30 03:46:52.147479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.147460 4442 scope.go:122] "RemoveContainer" containerID="4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd" Oct 30 03:46:52.147775 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:52.147736 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd\": container with ID starting with 4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd not found: ID does not exist" containerID="4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd" Oct 30 03:46:52.147832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.147774 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd"} err="failed to get container status \"4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd\": rpc error: code = NotFound desc = could not find container \"4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd\": container with ID starting with 4f9c12bcb95686cf04ac54165d2178fd27bebf13fe571459897c66caae54e9fd not found: ID does not exist" Oct 30 03:46:52.184825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.184804 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs"] Oct 30 03:46:52.186122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.186090 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:52.195152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.195128 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d5bf6f9f-dmvrs"] Oct 30 03:46:52.565471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.565438 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:52.961266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:52.961180 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:53.058994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.058962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:53.098038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.098004 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" podUID="3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" containerName="payload-processing" containerID="cri-o://7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e" gracePeriod=25 Oct 30 03:46:53.233684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.233651 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh"] Oct 30 03:46:53.234424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.234403 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" containerName="payload-processing" Oct 30 03:46:53.234424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.234424 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" containerName="payload-processing" Oct 30 03:46:53.234617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.234603 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" containerName="payload-processing" Oct 30 03:46:53.236983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.236960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:53.238182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.238167 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.271004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.270977 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh"] Oct 30 03:46:53.339409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339375 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339445 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8efd97-9c40-438c-9943-12236f7ac198-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339588 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339555 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjjbw\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-kube-api-access-mjjbw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339610 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339722 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339826 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339863 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339889 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8efd97-9c40-438c-9943-12236f7ac198-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.339989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.339921 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.419309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.419281 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:53.441021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.440982 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjjbw\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-kube-api-access-mjjbw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441040 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441108 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441150 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441167 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441198 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8efd97-9c40-438c-9943-12236f7ac198-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441222 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441275 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441361 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8efd97-9c40-438c-9943-12236f7ac198-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441457 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441508 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.441721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.441702 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.442123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.442066 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8efd97-9c40-438c-9943-12236f7ac198-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.442123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.442071 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.443843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.443628 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.444141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.444110 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8efd97-9c40-438c-9943-12236f7ac198-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.462350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.462321 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.462485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.462319 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjjbw\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-kube-api-access-mjjbw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-wgghh\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.527525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.527491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:53.542640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.542608 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-plugins-config-volume\") pod \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " Oct 30 03:46:53.542812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.542683 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-provider-credentials\") pod \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " Oct 30 03:46:53.542812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.542734 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-routing-overlay-volume\") pod \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " Oct 30 03:46:53.542812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.542791 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-kube-api-access-gz74t\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-kube-api-access-gz74t\") pod \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\" (UID: \"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5\") " Oct 30 03:46:53.543053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.542991 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-plugins-config-volume" pod "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" (UID: "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:53.543185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.543160 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-routing-overlay-volume" pod "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" (UID: "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:53.543394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.543378 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:53.543468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.543396 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:53.545498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.545475 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-provider-credentials" pod "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" (UID: "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:53.545585 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.545521 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-kube-api-access-gz74t" pod "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" (UID: "3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5"). InnerVolumeSpecName "kube-api-access-gz74t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:53.558922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.558900 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:53.645085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.645049 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz74t\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-kube-api-access-gz74t\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:53.645485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.645090 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:53.735471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.735442 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:53.906230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.906205 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh"] Oct 30 03:46:53.906441 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:46:53.906419 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb8efd97_9c40_438c_9943_12236f7ac198.slice/crio-7f078d5baf4272fe60bf3dc4b58533062e75498f5223380e704796fb9f86c9fa WatchSource:0}: Error finding container 7f078d5baf4272fe60bf3dc4b58533062e75498f5223380e704796fb9f86c9fa: Status 404 returned error can't find the container with id 7f078d5baf4272fe60bf3dc4b58533062e75498f5223380e704796fb9f86c9fa Oct 30 03:46:53.909250 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.909217 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:46:53.909322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.909288 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:46:53.909359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:53.909322 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:46:54.111158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.111123 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a6452d4-d4b3-4e11-ba0e-09929fec6bb3" path="/var/lib/kubelet/pods/7a6452d4-d4b3-4e11-ba0e-09929fec6bb3/volumes" Oct 30 03:46:54.155370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.155288 4442 generic.go:356] "Generic (PLEG): container finished" podID="3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" containerID="7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e" exitCode=0 Oct 30 03:46:54.155370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.155357 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" Oct 30 03:46:54.155597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.155375 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" event={"ID":"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5","Type":"ContainerDied","Data":"7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e"} Oct 30 03:46:54.155597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.155428 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst" event={"ID":"3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5","Type":"ContainerDied","Data":"8e111344224502cb1509460d0854483d0cb87af40b7d7b1dbadbb14517f5b37f"} Oct 30 03:46:54.155597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.155453 4442 scope.go:122] "RemoveContainer" containerID="7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e" Oct 30 03:46:54.157302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.157274 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" event={"ID":"eb8efd97-9c40-438c-9943-12236f7ac198","Type":"ContainerStarted","Data":"3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e"} Oct 30 03:46:54.157406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.157324 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" event={"ID":"eb8efd97-9c40-438c-9943-12236f7ac198","Type":"ContainerStarted","Data":"7f078d5baf4272fe60bf3dc4b58533062e75498f5223380e704796fb9f86c9fa"} Oct 30 03:46:54.167949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.167897 4442 scope.go:122] "RemoveContainer" containerID="7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e" Oct 30 03:46:54.168364 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:54.168329 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e\": container with ID starting with 7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e not found: ID does not exist" containerID="7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e" Oct 30 03:46:54.168458 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.168364 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e"} err="failed to get container status \"7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e\": rpc error: code = NotFound desc = could not find container \"7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e\": container with ID starting with 7f8e6f04da13797953dcc902b5fdbb79c3d22f1b7e099759ce788f174e75ab9e not found: ID does not exist" Oct 30 03:46:54.173993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.173965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:54.211368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.210742 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podStartSLOduration=1.210728082 podStartE2EDuration="1.210728082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:54.207965281 +0000 UTC m=+2638.210318479" watchObservedRunningTime="2026-10-30 03:46:54.210728082 +0000 UTC m=+2638.213081279" Oct 30 03:46:54.243091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.243062 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst"] Oct 30 03:46:54.248452 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.248428 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-6b86dfjst"] Oct 30 03:46:54.266862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.266839 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:54.561944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.561911 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:46:54.563968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.563939 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:46:54.564225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:54.564198 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:46:54.861883 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:54.861746 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:46:55.561244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:55.560289 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:46:55.561244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:55.560358 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:46:55.602780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:55.596823 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" podUID="03e57a0c-b697-41ec-b42a-c35f625668ba" containerName="payload-processing" containerID="cri-o://707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1" gracePeriod=25 Oct 30 03:46:55.826610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:55.826505 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:56.030775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.029136 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:56.116208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.113340 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-kube-api-access-c6jm6\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-kube-api-access-c6jm6\") pod \"03e57a0c-b697-41ec-b42a-c35f625668ba\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " Oct 30 03:46:56.116208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.113423 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-plugins-config-volume\") pod \"03e57a0c-b697-41ec-b42a-c35f625668ba\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " Oct 30 03:46:56.116208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.113521 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-provider-credentials\") pod \"03e57a0c-b697-41ec-b42a-c35f625668ba\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " Oct 30 03:46:56.116208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.113591 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-routing-overlay-volume\") pod \"03e57a0c-b697-41ec-b42a-c35f625668ba\" (UID: \"03e57a0c-b697-41ec-b42a-c35f625668ba\") " Oct 30 03:46:56.116208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.114348 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-routing-overlay-volume" pod "03e57a0c-b697-41ec-b42a-c35f625668ba" (UID: "03e57a0c-b697-41ec-b42a-c35f625668ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:56.116208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.114646 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-plugins-config-volume" pod "03e57a0c-b697-41ec-b42a-c35f625668ba" (UID: "03e57a0c-b697-41ec-b42a-c35f625668ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:46:56.116917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.116828 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-provider-credentials" pod "03e57a0c-b697-41ec-b42a-c35f625668ba" (UID: "03e57a0c-b697-41ec-b42a-c35f625668ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:56.117559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.117537 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-kube-api-access-c6jm6" pod "03e57a0c-b697-41ec-b42a-c35f625668ba" (UID: "03e57a0c-b697-41ec-b42a-c35f625668ba"). InnerVolumeSpecName "kube-api-access-c6jm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:46:56.123931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.123904 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" path="/var/lib/kubelet/pods/3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5/volumes" Oct 30 03:46:56.215960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.215915 4442 generic.go:356] "Generic (PLEG): container finished" podID="03e57a0c-b697-41ec-b42a-c35f625668ba" containerID="707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1" exitCode=0 Oct 30 03:46:56.216154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216011 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6jm6\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-kube-api-access-c6jm6\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:56.216154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216036 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:56.216154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216046 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/03e57a0c-b697-41ec-b42a-c35f625668ba-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:56.216154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216054 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/03e57a0c-b697-41ec-b42a-c35f625668ba-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:46:56.216154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216102 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" event={"ID":"03e57a0c-b697-41ec-b42a-c35f625668ba","Type":"ContainerDied","Data":"707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1"} Oct 30 03:46:56.216418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216169 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" event={"ID":"03e57a0c-b697-41ec-b42a-c35f625668ba","Type":"ContainerDied","Data":"9bdc4bac5f00b56f2b9bd24f5b8ca20a0f677cbc94f287785983ccb4f8bfe870"} Oct 30 03:46:56.216418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216180 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7" Oct 30 03:46:56.216418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.216189 4442 scope.go:122] "RemoveContainer" containerID="707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1" Oct 30 03:46:56.230919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.230894 4442 scope.go:122] "RemoveContainer" containerID="707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1" Oct 30 03:46:56.231258 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:46:56.231238 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1\": container with ID starting with 707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1 not found: ID does not exist" containerID="707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1" Oct 30 03:46:56.231357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.231266 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1"} err="failed to get container status \"707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1\": rpc error: code = NotFound desc = could not find container \"707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1\": container with ID starting with 707b46ff28d426f60405b5d8d4154944dccb90cc1e1c7ed24055e9836406a5a1 not found: ID does not exist" Oct 30 03:46:56.265046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.265018 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7"] Oct 30 03:46:56.268278 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.268258 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-67d95c9c8-n62m7"] Oct 30 03:46:56.523666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.523625 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:56.559890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.559860 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:46:56.560051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.559907 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:46:56.758245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.758196 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg"] Oct 30 03:46:56.758862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.758844 4442 cpu_manager.go:414] "containerMap: removing container" podUID="03e57a0c-b697-41ec-b42a-c35f625668ba" containerName="payload-processing" Oct 30 03:46:56.758862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.758860 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e57a0c-b697-41ec-b42a-c35f625668ba" containerName="payload-processing" Oct 30 03:46:56.759016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.758895 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" containerName="payload-processing" Oct 30 03:46:56.759016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.758901 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" containerName="payload-processing" Oct 30 03:46:56.759016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.758993 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6e31a0-e5e3-4ded-9b02-a8c2a5df37c5" containerName="payload-processing" Oct 30 03:46:56.759016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.759006 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e57a0c-b697-41ec-b42a-c35f625668ba" containerName="payload-processing" Oct 30 03:46:56.763818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.763800 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:56.779866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.779813 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"agc-cred-7b4014be3195\"" Oct 30 03:46:56.780027 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.780009 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"routing-overlay\"" Oct 30 03:46:56.780102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.780061 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"kube-root-ca.crt\"" Oct 30 03:46:56.780102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.780082 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-plugins-e2e-extmod-a-689bcf\"" Oct 30 03:46:56.780217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.780161 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-e2e--22d06fb9-dockercfg-kmzx9\"" Oct 30 03:46:56.780217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.780161 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"openshift-service-ca.crt\"" Oct 30 03:46:56.796784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.788881 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg"] Oct 30 03:46:56.936799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.931703 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:56.936799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.931847 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:56.936799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.931934 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fvpk\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-kube-api-access-4fvpk\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:56.936799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:56.932020 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.040840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.038053 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.040840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.038993 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fvpk\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-kube-api-access-4fvpk\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.040840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.039117 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.040840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.039283 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.040840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.040262 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.051801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.046190 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.073922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.072250 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.081685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.081662 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fvpk\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-kube-api-access-4fvpk\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.378024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.377938 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:57.539182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.539152 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:57.559632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.559587 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:46:57.559815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.559660 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:46:57.679312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.676730 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:57.755012 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.754708 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg"] Oct 30 03:46:57.760263 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:46:57.760031 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d20ac3c_12e5_4a6c_bb2a_e19807184269.slice/crio-983d976ae89b04bd49fefd091fe31d38f47105499e06bb05882aab776c7f4ff3 WatchSource:0}: Error finding container 983d976ae89b04bd49fefd091fe31d38f47105499e06bb05882aab776c7f4ff3: Status 404 returned error can't find the container with id 983d976ae89b04bd49fefd091fe31d38f47105499e06bb05882aab776c7f4ff3 Oct 30 03:46:57.820963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.818053 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h"] Oct 30 03:46:57.823745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.823717 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:57.831531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.831108 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h"] Oct 30 03:46:57.843984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.838349 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"agc-cred-299e9aafde73\"" Oct 30 03:46:57.855411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.854140 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"openshift-service-ca.crt\"" Oct 30 03:46:57.855411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.854278 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"payload-processing-external-model-e2e--154314f1-dockercfg-jhcf6\"" Oct 30 03:46:57.855411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.854382 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"kube-root-ca.crt\"" Oct 30 03:46:57.855411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.854479 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-35f43f\"" Oct 30 03:46:57.855411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.854588 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"routing-overlay\"" Oct 30 03:46:57.975848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.971922 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:57.975848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.971985 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:57.975848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.972115 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dxwg\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-kube-api-access-8dxwg\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:57.975848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:57.972230 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.073961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.073440 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dxwg\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-kube-api-access-8dxwg\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.073961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.073572 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.073961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.073687 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.073961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.073730 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.081075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.075367 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.081075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.075734 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.127298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.125425 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.127548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.127519 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dxwg\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-kube-api-access-8dxwg\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.135189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.135158 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e57a0c-b697-41ec-b42a-c35f625668ba" path="/var/lib/kubelet/pods/03e57a0c-b697-41ec-b42a-c35f625668ba/volumes" Oct 30 03:46:58.205070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.205036 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:58.238418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.238343 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" event={"ID":"4d20ac3c-12e5-4a6c-bb2a-e19807184269","Type":"ContainerStarted","Data":"b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5"} Oct 30 03:46:58.238591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.238566 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" event={"ID":"4d20ac3c-12e5-4a6c-bb2a-e19807184269","Type":"ContainerStarted","Data":"983d976ae89b04bd49fefd091fe31d38f47105499e06bb05882aab776c7f4ff3"} Oct 30 03:46:58.238670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.238618 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:46:58.514657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.514603 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" podStartSLOduration=2.514584424 podStartE2EDuration="2.514584424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:58.276564789 +0000 UTC m=+2642.278917987" watchObservedRunningTime="2026-10-30 03:46:58.514584424 +0000 UTC m=+2642.516937622" Oct 30 03:46:58.519610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.519586 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h"] Oct 30 03:46:58.559289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.559259 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:46:58.559412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:58.559319 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:46:59.227110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:59.227067 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:46:59.247072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:59.247036 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" event={"ID":"5852cb33-795c-4f71-b43f-342ee3ad8e12","Type":"ContainerStarted","Data":"5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126"} Oct 30 03:46:59.247235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:59.247102 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" event={"ID":"5852cb33-795c-4f71-b43f-342ee3ad8e12","Type":"ContainerStarted","Data":"2048fc49e5c21d3b4bb72a8ed7563fd7f04b08f278648746b60adcabd89a6485"} Oct 30 03:46:59.247322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:59.247305 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:46:59.559709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:59.559673 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:46:59.559909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:46:59.559727 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:47:00.559462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:00.559420 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:47:00.559947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:00.559479 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:47:00.886684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:00.886606 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:01.343773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.340255 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:01.440783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.437707 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:01.559322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.559282 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 03:47:01.559529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.559357 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 03:47:01.734215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.734121 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" podStartSLOduration=4.734107405 podStartE2EDuration="4.734107405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:59.334361279 +0000 UTC m=+2643.336714477" watchObservedRunningTime="2026-10-30 03:47:01.734107405 +0000 UTC m=+2645.736460605" Oct 30 03:47:01.734370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.734290 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg"] Oct 30 03:47:01.738388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.738369 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.755711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.755689 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg"] Oct 30 03:47:01.762392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.762264 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f0eb942fa2d9\"" Oct 30 03:47:01.762944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.762480 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 03:47:01.762944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.762580 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 03:47:01.762944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.762653 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 03:47:01.764973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.764936 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 03:47:01.764973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.764981 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pnpl6\"" Oct 30 03:47:01.809782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.809733 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-plugins-config-volume\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.810008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.809917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-provider-credentials\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.810102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.810028 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-routing-overlay-volume\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.810102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.810070 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wbr2\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-kube-api-access-4wbr2\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.913837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.913789 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-routing-overlay-volume\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.915033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.915007 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wbr2\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-kube-api-access-4wbr2\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.915276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.914921 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-routing-overlay-volume\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.915457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.915433 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-plugins-config-volume\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.915725 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.915704 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-provider-credentials\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.940795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.918918 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-plugins-config-volume\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.940795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.927622 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-provider-credentials\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:01.946208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:01.946174 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wbr2\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-kube-api-access-4wbr2\") pod \"payload-processing-external-model-54bdbcb9db-kxlgg\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:02.054854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:02.054811 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:02.381506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:02.381474 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg"] Oct 30 03:47:02.660943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:02.660850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:03.077432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.077403 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:03.142702 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.142663 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:47:03.143269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.143241 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:47:03.144268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.144248 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:47:03.178129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.178102 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:03.281416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.281381 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" event={"ID":"2364db73-8dfc-418f-bb52-528f19184198","Type":"ContainerStarted","Data":"440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193"} Oct 30 03:47:03.281745 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.281705 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:03.281925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.281786 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" event={"ID":"2364db73-8dfc-418f-bb52-528f19184198","Type":"ContainerStarted","Data":"76f18746cbcbb1e096c797fbecb144ff29a3e4540030f938506e83d1e6bce471"} Oct 30 03:47:03.325249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.325202 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" podStartSLOduration=2.325188568 podStartE2EDuration="2.325188568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:03.324347718 +0000 UTC m=+2647.326700916" watchObservedRunningTime="2026-10-30 03:47:03.325188568 +0000 UTC m=+2647.327541765" Oct 30 03:47:03.390118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.390003 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:03.700603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:03.700532 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:04.204934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:04.204898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:04.344142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:04.344098 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:04.450629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:04.450597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:05.763166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:05.763125 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:06.176298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:06.176208 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:06.736988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:06.736958 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:07.741171 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:07.741137 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:08.281838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.281787 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh"] Oct 30 03:47:08.289034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.288705 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.294719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.294685 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.294999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.294974 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.295158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.295139 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.295382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.295356 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.295521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.295492 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.295576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.295545 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.295614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.295600 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.295672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.295656 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfvv\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-kube-api-access-mdfvv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.295717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.295703 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.320205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.320176 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh"] Oct 30 03:47:08.396822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.396723 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.396822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.396823 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfvv\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-kube-api-access-mdfvv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.396868 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.396985 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397083 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397142 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397255 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397194 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397282 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397314 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397354 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397392 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397705 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.397736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.397717 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.398237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.398216 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.399685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.399664 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.399841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.399821 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.415063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.415036 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfvv\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-kube-api-access-mdfvv\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.415352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.415336 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-9p4jh\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.610368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.610307 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:08.926380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.926339 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh"] Oct 30 03:47:08.928276 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:47:08.928251 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04be625_d5ba_40b8_a4b2_cb4b80a0afaf.slice/crio-579db6c28c0ae17b5dfe4f2e0bfed616e646f9cd4741d7a2681e042f0b181877 WatchSource:0}: Error finding container 579db6c28c0ae17b5dfe4f2e0bfed616e646f9cd4741d7a2681e042f0b181877: Status 404 returned error can't find the container with id 579db6c28c0ae17b5dfe4f2e0bfed616e646f9cd4741d7a2681e042f0b181877 Oct 30 03:47:08.930949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.930920 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:08.931028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.930985 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:08.931028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:08.931022 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:09.248182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.248063 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:09.249789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.249767 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:47:09.359192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.359155 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" event={"ID":"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf","Type":"ContainerStarted","Data":"677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8"} Oct 30 03:47:09.359388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.359220 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" event={"ID":"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf","Type":"ContainerStarted","Data":"579db6c28c0ae17b5dfe4f2e0bfed616e646f9cd4741d7a2681e042f0b181877"} Oct 30 03:47:09.401646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.401586 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podStartSLOduration=1.4015673149999999 podStartE2EDuration="1.401567315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:09.396399881 +0000 UTC m=+2653.398753115" watchObservedRunningTime="2026-10-30 03:47:09.401567315 +0000 UTC m=+2653.403920513" Oct 30 03:47:09.610527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.610492 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:09.612438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.612413 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:09.612576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:09.612458 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:10.257348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:10.257319 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:47:10.611021 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:10.610942 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:10.611264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:10.611235 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:10.821348 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:10.820907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:11.546918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:11.546887 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:11.611705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:11.611668 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:11.611892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:11.611742 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:11.646705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:11.646666 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:12.614217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:12.614179 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:12.614827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:12.614794 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:13.076991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.076958 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:13.199782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.195441 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn"] Oct 30 03:47:13.222680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.222623 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn"] Oct 30 03:47:13.222894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.222735 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.236391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.236368 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-f29b9af32f32\"" Oct 30 03:47:13.236724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.236705 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-df757712a33a\"" Oct 30 03:47:13.247426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.247328 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-plugins-e2e-extmod-b-372480\"" Oct 30 03:47:13.247426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.247347 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"payload-processing-external-model-e2e--2cc204ae-dockercfg-mpmb2\"" Oct 30 03:47:13.247426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.247399 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"openshift-service-ca.crt\"" Oct 30 03:47:13.247677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.247472 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"routing-overlay\"" Oct 30 03:47:13.247677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.247631 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"kube-root-ca.crt\"" Oct 30 03:47:13.351193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.351112 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.351193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.351176 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.351397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.351298 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.351468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.351449 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxmc\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-kube-api-access-bxxmc\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.453038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.453007 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxmc\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-kube-api-access-bxxmc\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.453245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.453099 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.453245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.453132 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.453245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.453187 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.453784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.453718 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.453988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.453911 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.455857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.455833 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.470652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.470627 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxmc\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-kube-api-access-bxxmc\") pod \"payload-processing-external-model-e2e-extmod-b-372480-6db98stcn\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.547582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.547548 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:13.611166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.611073 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:13.611166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.611136 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:13.877492 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:47:13.877450 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d8dda4_21d6_4348_97db_8934f196dbf8.slice/crio-e2823b9b97f20c1aaae10e13a667b38f6506a93f50ac30f9507c7e2171c6971a WatchSource:0}: Error finding container e2823b9b97f20c1aaae10e13a667b38f6506a93f50ac30f9507c7e2171c6971a: Status 404 returned error can't find the container with id e2823b9b97f20c1aaae10e13a667b38f6506a93f50ac30f9507c7e2171c6971a Oct 30 03:47:13.878326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:13.878270 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn"] Oct 30 03:47:14.308398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.308370 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:14.429863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.428388 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" event={"ID":"79d8dda4-21d6-4348-97db-8934f196dbf8","Type":"ContainerStarted","Data":"34be0cad90eb09d419713c66a29bc0762c3ac6a308e6d0cac1abb3b7e0817c11"} Oct 30 03:47:14.429863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.428456 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" event={"ID":"79d8dda4-21d6-4348-97db-8934f196dbf8","Type":"ContainerStarted","Data":"e2823b9b97f20c1aaae10e13a667b38f6506a93f50ac30f9507c7e2171c6971a"} Oct 30 03:47:14.429863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.428862 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:14.468265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.468172 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" podStartSLOduration=1.46738807 podStartE2EDuration="1.46738807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:14.465910722 +0000 UTC m=+2658.468263936" watchObservedRunningTime="2026-10-30 03:47:14.46738807 +0000 UTC m=+2658.469741267" Oct 30 03:47:14.611782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.611676 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:14.611782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.611735 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:14.711235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:14.711210 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:15.611276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:15.611234 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:15.611821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:15.611283 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:16.230789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:16.230697 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:16.610743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:16.610706 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 03:47:16.611138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:16.611107 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 03:47:18.156699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:18.156658 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:18.576792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:18.576743 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:18.577233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:18.577204 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:18.578077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:18.578058 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:47:19.196880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:19.196839 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-wgghh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:47:19.197516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:19.197482 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:47:19.912662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:19.912626 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:21.377997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:21.377963 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:22.797630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:22.797600 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:24.183472 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:24.183433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:24.868994 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:47:24.868952 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:47:25.438424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:25.438386 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:47:25.626402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:25.626370 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:27.139274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:27.139233 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:28.601690 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:28.601658 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:30.091271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:30.091239 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:30.943438 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:30.943398 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg"] Oct 30 03:47:30.956801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:30.956768 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:31.052369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:31.052339 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:31.846681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:31.846647 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:33.164219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:33.164187 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:33.264140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:33.264108 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:33.451654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:33.451527 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:34.685424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:34.685391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:35.304464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:35.304425 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:35.943854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:35.943815 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" podUID="2364db73-8dfc-418f-bb52-528f19184198" containerName="payload-processing" containerID="cri-o://440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193" gracePeriod=25 Oct 30 03:47:36.205812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.205739 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:36.252064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.250830 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-plugins-config-volume\") pod \"2364db73-8dfc-418f-bb52-528f19184198\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " Oct 30 03:47:36.252064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.250905 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-routing-overlay-volume\") pod \"2364db73-8dfc-418f-bb52-528f19184198\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " Oct 30 03:47:36.252064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.250974 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-provider-credentials\") pod \"2364db73-8dfc-418f-bb52-528f19184198\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " Oct 30 03:47:36.252064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.251028 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-kube-api-access-4wbr2\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-kube-api-access-4wbr2\") pod \"2364db73-8dfc-418f-bb52-528f19184198\" (UID: \"2364db73-8dfc-418f-bb52-528f19184198\") " Oct 30 03:47:36.253703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.252898 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-plugins-config-volume" pod "2364db73-8dfc-418f-bb52-528f19184198" (UID: "2364db73-8dfc-418f-bb52-528f19184198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:36.254212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.254184 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-routing-overlay-volume" pod "2364db73-8dfc-418f-bb52-528f19184198" (UID: "2364db73-8dfc-418f-bb52-528f19184198"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:36.262244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.262206 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-kube-api-access-4wbr2" pod "2364db73-8dfc-418f-bb52-528f19184198" (UID: "2364db73-8dfc-418f-bb52-528f19184198"). InnerVolumeSpecName "kube-api-access-4wbr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:36.267009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.266984 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-provider-credentials" pod "2364db73-8dfc-418f-bb52-528f19184198" (UID: "2364db73-8dfc-418f-bb52-528f19184198"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:36.353259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.353228 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:36.353259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.353255 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:36.353476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.353275 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wbr2\" (UniqueName: \"kubernetes.io/projected/2364db73-8dfc-418f-bb52-528f19184198-kube-api-access-4wbr2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:36.353476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.353283 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2364db73-8dfc-418f-bb52-528f19184198-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:36.663475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.663437 4442 generic.go:356] "Generic (PLEG): container finished" podID="2364db73-8dfc-418f-bb52-528f19184198" containerID="440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193" exitCode=0 Oct 30 03:47:36.663662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.663489 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" Oct 30 03:47:36.663662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.663508 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" event={"ID":"2364db73-8dfc-418f-bb52-528f19184198","Type":"ContainerDied","Data":"440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193"} Oct 30 03:47:36.663662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.663560 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg" event={"ID":"2364db73-8dfc-418f-bb52-528f19184198","Type":"ContainerDied","Data":"76f18746cbcbb1e096c797fbecb144ff29a3e4540030f938506e83d1e6bce471"} Oct 30 03:47:36.663662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.663589 4442 scope.go:122] "RemoveContainer" containerID="440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193" Oct 30 03:47:36.674064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.674045 4442 scope.go:122] "RemoveContainer" containerID="440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193" Oct 30 03:47:36.674325 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:47:36.674305 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193\": container with ID starting with 440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193 not found: ID does not exist" containerID="440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193" Oct 30 03:47:36.674380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.674334 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193"} err="failed to get container status \"440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193\": rpc error: code = NotFound desc = could not find container \"440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193\": container with ID starting with 440da058987f2020ca3d106aae7e8f5518c43a3c57771c57801cc3f3fa803193 not found: ID does not exist" Oct 30 03:47:36.711071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.711036 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg"] Oct 30 03:47:36.721741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.721717 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54bdbcb9db-kxlgg"] Oct 30 03:47:36.890610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:36.890586 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:38.107890 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:38.107858 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2364db73-8dfc-418f-bb52-528f19184198" path="/var/lib/kubelet/pods/2364db73-8dfc-418f-bb52-528f19184198/volumes" Oct 30 03:47:38.502173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:38.502144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:39.476385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:39.476351 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:39.572921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:39.572887 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:40.217640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:40.217605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:40.628508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:40.628479 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:40.729845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:40.729811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:41.623567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.623539 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:41.907640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.907544 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-tjkvv"] Oct 30 03:47:41.908249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.908226 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2364db73-8dfc-418f-bb52-528f19184198" containerName="payload-processing" Oct 30 03:47:41.908249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.908244 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2364db73-8dfc-418f-bb52-528f19184198" containerName="payload-processing" Oct 30 03:47:41.908548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.908423 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2364db73-8dfc-418f-bb52-528f19184198" containerName="payload-processing" Oct 30 03:47:41.912265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.912249 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:41.927826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.927809 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7e23b6c624f1\"" Oct 30 03:47:41.942546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.942506 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tz8wv\"" Oct 30 03:47:41.942698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.942557 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 03:47:41.942698 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.942559 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 03:47:41.942875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.942696 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 03:47:41.945933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.945913 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 03:47:41.947499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.947475 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-plugins-config-volume\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:41.947626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.947534 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-routing-overlay-volume\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:41.947626 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.947575 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79zz\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-kube-api-access-w79zz\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:41.947796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.947648 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-provider-credentials\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:41.948344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:41.948321 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-tjkvv"] Oct 30 03:47:42.083777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.069284 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-provider-credentials\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.083777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.069543 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-plugins-config-volume\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.083777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.069593 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-routing-overlay-volume\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.083777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.069639 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79zz\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-kube-api-access-w79zz\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.083777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.076439 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-plugins-config-volume\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.083777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.077127 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-routing-overlay-volume\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.105535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.101227 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-provider-credentials\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.110561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.110527 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79zz\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-kube-api-access-w79zz\") pod \"payload-processing-external-model-5bc757568-tjkvv\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.226101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.226003 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.258546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.258511 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:42.602875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.602849 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-tjkvv"] Oct 30 03:47:42.603509 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:47:42.603479 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9214efd1_44ed_4575_9507_c1508914fb96.slice/crio-24157cc0e6aae923ce01cdc824aa6ead12a118c6f2b7f40823fc88c0b46d920b WatchSource:0}: Error finding container 24157cc0e6aae923ce01cdc824aa6ead12a118c6f2b7f40823fc88c0b46d920b: Status 404 returned error can't find the container with id 24157cc0e6aae923ce01cdc824aa6ead12a118c6f2b7f40823fc88c0b46d920b Oct 30 03:47:42.716001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.715969 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" event={"ID":"9214efd1-44ed-4575-9507-c1508914fb96","Type":"ContainerStarted","Data":"a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95"} Oct 30 03:47:42.716154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.716029 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" event={"ID":"9214efd1-44ed-4575-9507-c1508914fb96","Type":"ContainerStarted","Data":"24157cc0e6aae923ce01cdc824aa6ead12a118c6f2b7f40823fc88c0b46d920b"} Oct 30 03:47:42.716154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.716054 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:42.754069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:42.754022 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" podStartSLOduration=1.754008647 podStartE2EDuration="1.754008647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:42.751569853 +0000 UTC m=+2686.753923031" watchObservedRunningTime="2026-10-30 03:47:42.754008647 +0000 UTC m=+2686.756361843" Oct 30 03:47:43.280295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:43.280256 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:43.393379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:43.393344 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:43.889324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:43.889287 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:44.404137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:44.404100 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:44.501006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:44.500958 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:45.018400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:45.018367 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:45.600399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:45.600364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:46.996972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:46.996941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:47.068554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:47.068522 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:47:47.140051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:47.140019 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-tjkvv"] Oct 30 03:47:47.143194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:47.143167 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:47.208743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:47.208712 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:47.311274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:47.311242 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:48.684419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:48.684364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:48.822931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:48.822884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:49.274166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:49.274134 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:49.414773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:49.405566 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:49.837664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:49.837631 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:50.509279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:50.509247 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:50.776059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:50.775977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:51.637173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:51.637137 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:51.744867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:51.744835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:51.835954 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:51.835921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:52.140688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.140642 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" podUID="9214efd1-44ed-4575-9507-c1508914fb96" containerName="payload-processing" containerID="cri-o://a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95" gracePeriod=25 Oct 30 03:47:52.235721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.235694 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:52.444254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.444222 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:52.518201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.518170 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-plugins-config-volume\") pod \"9214efd1-44ed-4575-9507-c1508914fb96\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " Oct 30 03:47:52.518404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.518226 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-routing-overlay-volume\") pod \"9214efd1-44ed-4575-9507-c1508914fb96\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " Oct 30 03:47:52.518404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.518301 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-kube-api-access-w79zz\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-kube-api-access-w79zz\") pod \"9214efd1-44ed-4575-9507-c1508914fb96\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " Oct 30 03:47:52.518522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.518404 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-provider-credentials\") pod \"9214efd1-44ed-4575-9507-c1508914fb96\" (UID: \"9214efd1-44ed-4575-9507-c1508914fb96\") " Oct 30 03:47:52.518696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.518633 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-plugins-config-volume" pod "9214efd1-44ed-4575-9507-c1508914fb96" (UID: "9214efd1-44ed-4575-9507-c1508914fb96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:52.518799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.518714 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-routing-overlay-volume" pod "9214efd1-44ed-4575-9507-c1508914fb96" (UID: "9214efd1-44ed-4575-9507-c1508914fb96"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:47:52.519152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.519125 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:52.519152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.519156 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9214efd1-44ed-4575-9507-c1508914fb96-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:52.521135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.521108 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-kube-api-access-w79zz" pod "9214efd1-44ed-4575-9507-c1508914fb96" (UID: "9214efd1-44ed-4575-9507-c1508914fb96"). InnerVolumeSpecName "kube-api-access-w79zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:52.527172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.527131 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-provider-credentials" pod "9214efd1-44ed-4575-9507-c1508914fb96" (UID: "9214efd1-44ed-4575-9507-c1508914fb96"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:47:52.620037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.620003 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w79zz\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-kube-api-access-w79zz\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:52.620037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.620030 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9214efd1-44ed-4575-9507-c1508914fb96-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:47:52.757939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.757904 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:52.891544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.891512 4442 generic.go:356] "Generic (PLEG): container finished" podID="9214efd1-44ed-4575-9507-c1508914fb96" containerID="a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95" exitCode=0 Oct 30 03:47:52.891727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.891563 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" Oct 30 03:47:52.891727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.891603 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" event={"ID":"9214efd1-44ed-4575-9507-c1508914fb96","Type":"ContainerDied","Data":"a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95"} Oct 30 03:47:52.891727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.891658 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bc757568-tjkvv" event={"ID":"9214efd1-44ed-4575-9507-c1508914fb96","Type":"ContainerDied","Data":"24157cc0e6aae923ce01cdc824aa6ead12a118c6f2b7f40823fc88c0b46d920b"} Oct 30 03:47:52.891727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.891686 4442 scope.go:122] "RemoveContainer" containerID="a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95" Oct 30 03:47:52.902705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.902689 4442 scope.go:122] "RemoveContainer" containerID="a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95" Oct 30 03:47:52.902983 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:47:52.902964 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95\": container with ID starting with a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95 not found: ID does not exist" containerID="a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95" Oct 30 03:47:52.903046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.902992 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95"} err="failed to get container status \"a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95\": rpc error: code = NotFound desc = could not find container \"a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95\": container with ID starting with a5b30337564566f2707d847d173631d934220948ed46984df369bf577e61cd95 not found: ID does not exist" Oct 30 03:47:52.938903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.938883 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-tjkvv"] Oct 30 03:47:52.942091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:52.942070 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bc757568-tjkvv"] Oct 30 03:47:53.401204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.401165 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl"] Oct 30 03:47:53.402591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.402556 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9214efd1-44ed-4575-9507-c1508914fb96" containerName="payload-processing" Oct 30 03:47:53.402591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.402589 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9214efd1-44ed-4575-9507-c1508914fb96" containerName="payload-processing" Oct 30 03:47:53.403032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.403012 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9214efd1-44ed-4575-9507-c1508914fb96" containerName="payload-processing" Oct 30 03:47:53.408058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.408037 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.423577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.423547 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl"] Oct 30 03:47:53.533667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.533631 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.533667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.533666 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.533933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.533689 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.533933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.533883 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czjzc\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-kube-api-access-czjzc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.534004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.533964 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.534042 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.534003 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.534089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.534070 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.534175 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.534154 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.534347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.534330 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.635971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.635938 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.635981 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.635998 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636018 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636064 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czjzc\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-kube-api-access-czjzc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636115 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636148 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636245 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636275 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636557 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636513 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636625 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636633 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.636913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.636802 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.637270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.637249 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.638873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.638847 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.638973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.638872 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.655391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.655329 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czjzc\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-kube-api-access-czjzc\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.656566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.656544 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-tvsnl\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.724286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.724254 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:53.736406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.736379 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:53.836464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:53.836423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:54.057519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:54.057479 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl"] Oct 30 03:47:54.057718 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:47:54.057697 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79de6e0_ad3d_4cd7_9bde_8cdc5633ed8b.slice/crio-e3c969f418fd792566832711c92562dcd4469aaffdd79832d43bd221205aa23f WatchSource:0}: Error finding container e3c969f418fd792566832711c92562dcd4469aaffdd79832d43bd221205aa23f: Status 404 returned error can't find the container with id e3c969f418fd792566832711c92562dcd4469aaffdd79832d43bd221205aa23f Oct 30 03:47:54.060496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:54.060465 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:54.060633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:54.060544 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:54.060633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:54.060591 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:54.108697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:54.108669 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9214efd1-44ed-4575-9507-c1508914fb96" path="/var/lib/kubelet/pods/9214efd1-44ed-4575-9507-c1508914fb96/volumes" Oct 30 03:47:54.862793 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:47:54.862729 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:47:54.935433 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:54.935395 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" event={"ID":"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b","Type":"ContainerStarted","Data":"25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c"} Oct 30 03:47:54.935640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:54.935447 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" event={"ID":"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b","Type":"ContainerStarted","Data":"e3c969f418fd792566832711c92562dcd4469aaffdd79832d43bd221205aa23f"} Oct 30 03:47:55.010067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:55.010005 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podStartSLOduration=2.00998759 podStartE2EDuration="2.00998759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:55.005471822 +0000 UTC m=+2699.007825016" watchObservedRunningTime="2026-10-30 03:47:55.00998759 +0000 UTC m=+2699.012340789" Oct 30 03:47:55.412324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:55.412286 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:55.725488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:55.725395 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:47:55.726942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:55.726908 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:47:55.727069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:55.726970 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:47:56.259084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.259006 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72"] Oct 30 03:47:56.263636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.263615 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.277050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.277030 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e10c0f-openshift-default-dockercfg-gj6dh\"" Oct 30 03:47:56.321625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.321585 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72"] Oct 30 03:47:56.407718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.407677 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:56.415944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.415913 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-credential-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.415959 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-envoy\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.415981 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-certs\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.416115 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-istio-token\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.416141 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-data\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.416166 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff725a5-955f-433a-a175-019d98942e1e-istiod-ca-cert\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.416260 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.416312 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84jn\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-kube-api-access-l84jn\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.416362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.416343 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff725a5-955f-433a-a175-019d98942e1e-istio-podinfo\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.508781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.507905 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:56.517399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517319 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-istio-token\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.517399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517384 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-data\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.517709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517412 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff725a5-955f-433a-a175-019d98942e1e-istiod-ca-cert\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.517709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517469 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.517709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84jn\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-kube-api-access-l84jn\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.517709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517526 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff725a5-955f-433a-a175-019d98942e1e-istio-podinfo\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.517709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517699 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-credential-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.518054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517742 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-envoy\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.518054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517791 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-certs\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.518054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.517924 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.518245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.518063 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-certs\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.518375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.518351 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-credential-socket\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.518700 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.518652 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff725a5-955f-433a-a175-019d98942e1e-istiod-ca-cert\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.519059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.519036 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-data\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.521087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.521062 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-envoy\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.521383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.521356 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff725a5-955f-433a-a175-019d98942e1e-istio-podinfo\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.536709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.536666 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-istio-token\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.537042 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.537023 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84jn\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-kube-api-access-l84jn\") pod \"e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.537584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.537565 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:47:56.540038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.540019 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:47:56.578659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.578639 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:56.823829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.822931 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:47:56.823829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:56.822995 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:47:57.213877 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.213835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:57.213981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.213932 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72"] Oct 30 03:47:57.215888 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:47:57.215867 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff725a5_955f_433a_a175_019d98942e1e.slice/crio-0e339c0040d56c531ad87baa6a7bab6fa75cf3293c248007ef9aa8a8344b4882 WatchSource:0}: Error finding container 0e339c0040d56c531ad87baa6a7bab6fa75cf3293c248007ef9aa8a8344b4882: Status 404 returned error can't find the container with id 0e339c0040d56c531ad87baa6a7bab6fa75cf3293c248007ef9aa8a8344b4882 Oct 30 03:47:57.218551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.218519 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:57.218641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.218586 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:57.218641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.218617 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:47:57.735821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.735777 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:47:57.736399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.735846 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:47:57.896786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:57.895124 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:58.017414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.017263 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:58.022088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.022056 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" event={"ID":"dff725a5-955f-433a-a175-019d98942e1e","Type":"ContainerStarted","Data":"a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8"} Oct 30 03:47:58.022248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.022146 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" event={"ID":"dff725a5-955f-433a-a175-019d98942e1e","Type":"ContainerStarted","Data":"0e339c0040d56c531ad87baa6a7bab6fa75cf3293c248007ef9aa8a8344b4882"} Oct 30 03:47:58.113263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.113207 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podStartSLOduration=2.113189478 podStartE2EDuration="2.113189478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:47:58.071111272 +0000 UTC m=+2702.073464473" watchObservedRunningTime="2026-10-30 03:47:58.113189478 +0000 UTC m=+2702.115542739" Oct 30 03:47:58.409830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.407590 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:58.579834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.579791 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:47:58.581361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.581332 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 03:47:58.581499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.581379 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 03:47:58.733339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.733231 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:47:58.733339 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.733300 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:47:58.881546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.881500 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:58.981117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:58.981081 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:59.078701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:59.078667 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:59.188779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:59.183859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:47:59.579605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:59.579571 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 03:47:59.579796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:59.579624 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 03:47:59.727898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:59.727864 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:47:59.728292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:47:59.728263 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:48:00.579980 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:00.579943 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 03:48:00.580409 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:00.579996 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 03:48:00.734779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:00.734126 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:48:00.734779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:00.734184 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:48:00.811778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:00.810440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:01.581001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:01.580963 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 03:48:01.581453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:01.581031 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 03:48:01.725475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:01.725435 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:48:01.725684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:01.725487 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:48:02.267379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.267340 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn"] Oct 30 03:48:02.275084 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.275055 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.308408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.305087 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-372480-openshift-default-dockercfg-564ss\"" Oct 30 03:48:02.336078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.311558 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn"] Oct 30 03:48:02.336078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.335629 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.335700 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87aded6a-1932-4856-ae97-c18293a8dec6-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.335741 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.335803 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf6d\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-kube-api-access-zgf6d\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.336110 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.336148 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.336173 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.336374 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.336525 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.336426 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87aded6a-1932-4856-ae97-c18293a8dec6-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.437997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.437951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438019 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87aded6a-1932-4856-ae97-c18293a8dec6-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438130 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438220 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87aded6a-1932-4856-ae97-c18293a8dec6-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438278 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438305 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf6d\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-kube-api-access-zgf6d\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438397 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438427 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438451 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.438704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.438676 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-data\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.439050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.439021 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.439197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.439078 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-certs\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.439197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.439084 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-credential-socket\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.439197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.439124 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87aded6a-1932-4856-ae97-c18293a8dec6-istiod-ca-cert\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.441278 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.441251 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-envoy\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.441461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.441437 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87aded6a-1932-4856-ae97-c18293a8dec6-istio-podinfo\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.457958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.457918 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf6d\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-kube-api-access-zgf6d\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.458043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.457960 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-istio-token\") pod \"e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.579779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.579679 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 03:48:02.579779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.579737 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 03:48:02.636414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.636378 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:02.711302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.710281 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:02.732562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.732114 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 03:48:02.732562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.732173 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 03:48:02.959931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:02.959706 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:03.044991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.043595 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn"] Oct 30 03:48:03.050835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.050026 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:48:03.050835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.050112 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:48:03.050835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.050153 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:48:03.060407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.060376 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:03.123699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.123618 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" event={"ID":"87aded6a-1932-4856-ae97-c18293a8dec6","Type":"ContainerStarted","Data":"7e7534f7ee53ccfa5bbbd15a3e1b79e18ba47bd6d4f2141bf19c0116cc1feddf"} Oct 30 03:48:03.581310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.581270 4442 patch_prober.go:29] interesting pod/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 03:48:03.581546 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:03.581333 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 03:48:04.024132 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.024099 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:48:04.026347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.026322 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:48:04.033134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.033102 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:48:04.138847 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.138774 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" event={"ID":"87aded6a-1932-4856-ae97-c18293a8dec6","Type":"ContainerStarted","Data":"6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7"} Oct 30 03:48:04.179686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.179626 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podStartSLOduration=2.17960658 podStartE2EDuration="2.17960658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:04.178919563 +0000 UTC m=+2708.181272758" watchObservedRunningTime="2026-10-30 03:48:04.17960658 +0000 UTC m=+2708.181959780" Oct 30 03:48:04.597778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.594094 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:48:04.597778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.596241 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:48:04.608030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.607993 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 03:48:04.614632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.614581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:04.637729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.637705 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:04.638806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.638781 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 03:48:04.638934 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.638831 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 03:48:04.758267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.758236 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:04.862523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:04.862453 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:05.637544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:05.637511 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 03:48:05.637972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:05.637561 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 03:48:06.319846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:06.319811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:06.637824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:06.637726 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 03:48:06.637824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:06.637794 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 03:48:07.637505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:07.637466 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 03:48:07.637693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:07.637523 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 03:48:07.880253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:07.880216 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:08.637375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:08.637317 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 03:48:08.637548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:08.637430 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 03:48:09.482550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:09.482517 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:09.637195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:09.637150 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 03:48:09.637404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:09.637212 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 03:48:10.642479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:10.642447 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:10.643004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:10.642844 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:10.644079 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:10.644055 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 03:48:10.912548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:10.912472 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:12.481724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:12.481694 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:13.970487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:13.970457 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:15.401282 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:15.401251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:16.905372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:16.905329 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:18.655433 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:18.655365 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:19.633778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:19.630838 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:48:19.633778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:19.630931 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:48:20.070942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:20.070903 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:48:20.071364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:20.070987 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:48:20.399905 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:20.396428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:21.791045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:21.790996 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:23.270775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:23.270728 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:24.802113 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:24.802072 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:24.909543 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:48:24.909503 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:48:26.305502 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:26.305469 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:27.788151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:27.788111 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:29.196480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:29.196440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:30.573399 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:30.573362 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:31.429133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:31.429099 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:48:31.992376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:31.992348 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:32.074076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:32.074028 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:32.181822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:32.181782 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:33.505949 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:33.505914 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:34.988430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:34.988397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:36.370212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:36.370167 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:37.802451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:37.802415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:39.261882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:39.261789 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:40.763723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:40.763690 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:40.883237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:40.883204 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:42.419544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:42.419514 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:44.101073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:44.101041 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:45.596192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:45.596141 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:47.260977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:47.260943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:48.922693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:48.922658 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:49.952767 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:49.952725 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:50.384435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:50.384397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:50.895165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:50.895132 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:51.923408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:51.923364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:52.896178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:52.896149 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:53.501722 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:53.501657 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:54.862109 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:48:54.862076 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:48:55.027300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:55.027269 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:56.504900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:56.504864 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:58.037800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:58.030314 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:48:59.991246 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:48:59.991205 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:01.511937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:01.511899 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:02.994936 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:02.994900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:04.609263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:04.609232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:06.201978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:06.201900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:07.682177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:07.682138 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:09.215942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:09.215889 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:10.801208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:10.801176 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:12.366307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:12.366271 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:13.990475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:13.990437 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:15.235868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:15.235835 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:49:15.587334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:15.587308 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:15.689280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:15.689243 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:16.080741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:16.080704 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:16.547549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:16.547506 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:16.716263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:16.716228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:17.393643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:17.393597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:19.025990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:19.025952 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:19.574001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:19.573965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:19.672836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:19.672785 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:21.239665 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.239618 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:21.305515 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.305483 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4"] Oct 30 03:49:21.309989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.309968 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.335528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.335505 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-35f43f\"/\"agc-cred-0b9302668378\"" Oct 30 03:49:21.346378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.346349 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4"] Oct 30 03:49:21.387163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.387137 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:21.450778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.450706 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.451148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.451125 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.453555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.453530 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.453819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.453797 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5cbp\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-kube-api-access-f5cbp\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.475994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.475962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:21.554791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.554727 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.554972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.554844 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5cbp\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-kube-api-access-f5cbp\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.555066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.554999 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.555066 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.555055 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.555677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.555650 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.555821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.555739 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.557510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.557485 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.572915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.572892 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5cbp\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-kube-api-access-f5cbp\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.624178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.624149 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:21.927590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:21.927564 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4"] Oct 30 03:49:21.927723 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:49:21.927585 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbcf37d_25a6_4a04_9863_974293babad6.slice/crio-caf478e56abdbd75bc7027e3feabca5026413e44042bdb494fbd4580fa2b5340 WatchSource:0}: Error finding container caf478e56abdbd75bc7027e3feabca5026413e44042bdb494fbd4580fa2b5340: Status 404 returned error can't find the container with id caf478e56abdbd75bc7027e3feabca5026413e44042bdb494fbd4580fa2b5340 Oct 30 03:49:22.001719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:22.001686 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" event={"ID":"ccbcf37d-25a6-4a04-9863-974293babad6","Type":"ContainerStarted","Data":"494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051"} Oct 30 03:49:22.001872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:22.001765 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" event={"ID":"ccbcf37d-25a6-4a04-9863-974293babad6","Type":"ContainerStarted","Data":"caf478e56abdbd75bc7027e3feabca5026413e44042bdb494fbd4580fa2b5340"} Oct 30 03:49:22.001872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:22.001795 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:22.041402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:22.041354 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" podStartSLOduration=1.041339735 podStartE2EDuration="1.041339735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:22.037715284 +0000 UTC m=+2786.040068493" watchObservedRunningTime="2026-10-30 03:49:22.041339735 +0000 UTC m=+2786.043692972" Oct 30 03:49:23.056903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:23.056852 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:23.159584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:23.159525 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:24.633446 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:24.633414 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:24.733811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:24.733778 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:24.877791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:24.861461 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:24.944070 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:49:24.943976 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:49:25.875272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:25.875237 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:25.959707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:25.959677 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:26.543059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:26.543015 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:26.701173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:26.701142 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:28.200852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:28.200820 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:28.815151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:28.815116 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf"] Oct 30 03:49:28.876533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:28.876491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:29.683015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:29.682983 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:29.854161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:29.854131 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:30.326470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:30.326426 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:30.799746 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:30.799698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:31.038134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:31.038097 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:31.504030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:31.503995 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:33.012135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:33.012107 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:49:33.092480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:33.092448 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h"] Oct 30 03:49:33.174295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:33.174264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:33.360222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:33.360130 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:33.468325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:33.468292 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:33.551381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:33.551350 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:33.816301 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:33.816262 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" podUID="62ec0153-3f7d-46ed-8602-4ca47d5b327d" containerName="payload-processing" containerID="cri-o://d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004" gracePeriod=25 Oct 30 03:49:34.093727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.093700 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:49:34.141031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.141004 4442 generic.go:356] "Generic (PLEG): container finished" podID="62ec0153-3f7d-46ed-8602-4ca47d5b327d" containerID="d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004" exitCode=0 Oct 30 03:49:34.141178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.141053 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" Oct 30 03:49:34.141178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.141095 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" event={"ID":"62ec0153-3f7d-46ed-8602-4ca47d5b327d","Type":"ContainerDied","Data":"d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004"} Oct 30 03:49:34.141178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.141145 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf" event={"ID":"62ec0153-3f7d-46ed-8602-4ca47d5b327d","Type":"ContainerDied","Data":"2636dda0c90bab9e2ab80263f7ac1ba3ea97c7bab1ff46082fbfecd087a91f53"} Oct 30 03:49:34.141178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.141177 4442 scope.go:122] "RemoveContainer" containerID="d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004" Oct 30 03:49:34.153783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.153731 4442 scope.go:122] "RemoveContainer" containerID="d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004" Oct 30 03:49:34.154130 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:49:34.154102 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004\": container with ID starting with d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004 not found: ID does not exist" containerID="d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004" Oct 30 03:49:34.154217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.154143 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004"} err="failed to get container status \"d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004\": rpc error: code = NotFound desc = could not find container \"d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004\": container with ID starting with d86ecbf4ef0402c266578ceb704f7f6d4f44a7eab786286ac1c5745743571004 not found: ID does not exist" Oct 30 03:49:34.228273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.228240 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-provider-credentials\") pod \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " Oct 30 03:49:34.228483 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.228376 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-routing-overlay-volume\") pod \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " Oct 30 03:49:34.228574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.228483 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-plugins-config-volume\") pod \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " Oct 30 03:49:34.228675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.228579 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-kube-api-access-9wjjm\") pod \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\" (UID: \"62ec0153-3f7d-46ed-8602-4ca47d5b327d\") " Oct 30 03:49:34.228861 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.228825 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-routing-overlay-volume" pod "62ec0153-3f7d-46ed-8602-4ca47d5b327d" (UID: "62ec0153-3f7d-46ed-8602-4ca47d5b327d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:34.229004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.228982 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-plugins-config-volume" pod "62ec0153-3f7d-46ed-8602-4ca47d5b327d" (UID: "62ec0153-3f7d-46ed-8602-4ca47d5b327d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:34.229567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.229549 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:34.229674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.229572 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62ec0153-3f7d-46ed-8602-4ca47d5b327d-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:34.231325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.231302 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-kube-api-access-9wjjm" pod "62ec0153-3f7d-46ed-8602-4ca47d5b327d" (UID: "62ec0153-3f7d-46ed-8602-4ca47d5b327d"). InnerVolumeSpecName "kube-api-access-9wjjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:49:34.231415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.231339 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-provider-credentials" pod "62ec0153-3f7d-46ed-8602-4ca47d5b327d" (UID: "62ec0153-3f7d-46ed-8602-4ca47d5b327d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:49:34.330922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.330844 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wjjm\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-kube-api-access-9wjjm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:34.330922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.330875 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62ec0153-3f7d-46ed-8602-4ca47d5b327d-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:34.495102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.495069 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf"] Oct 30 03:49:34.499284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.499249 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-441461/payload-processing-external-model-e2e-extmod-a-441461-7447ph7zf"] Oct 30 03:49:34.632478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.632391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:34.779038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.779001 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:34.878087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:34.878043 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:35.025378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:35.025349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:35.900605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:35.900563 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg"] Oct 30 03:49:36.109300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.109264 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ec0153-3f7d-46ed-8602-4ca47d5b327d" path="/var/lib/kubelet/pods/62ec0153-3f7d-46ed-8602-4ca47d5b327d/volumes" Oct 30 03:49:36.434445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.434413 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:36.534237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.534206 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:36.556606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.556567 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529"] Oct 30 03:49:36.557395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.557373 4442 cpu_manager.go:414] "containerMap: removing container" podUID="62ec0153-3f7d-46ed-8602-4ca47d5b327d" containerName="payload-processing" Oct 30 03:49:36.557395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.557396 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ec0153-3f7d-46ed-8602-4ca47d5b327d" containerName="payload-processing" Oct 30 03:49:36.557573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.557541 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ec0153-3f7d-46ed-8602-4ca47d5b327d" containerName="payload-processing" Oct 30 03:49:36.562985 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.562966 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:36.580570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.580545 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529"] Oct 30 03:49:36.587092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.587061 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-35fc87\"" Oct 30 03:49:36.587264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.587249 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-35fc87-dockercfg-lh6t9\"" Oct 30 03:49:36.663829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.663070 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e77fc9f-bada-453e-8b68-b57512b8850b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:36.663829 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.663291 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gdff\" (UniqueName: \"kubernetes.io/projected/5e77fc9f-bada-453e-8b68-b57512b8850b-kube-api-access-8gdff\") pod \"payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:36.820776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.807824 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gdff\" (UniqueName: \"kubernetes.io/projected/5e77fc9f-bada-453e-8b68-b57512b8850b-kube-api-access-8gdff\") pod \"payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:36.820776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.808081 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e77fc9f-bada-453e-8b68-b57512b8850b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:36.820776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.809084 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e77fc9f-bada-453e-8b68-b57512b8850b-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:36.846680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.845894 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:36.862823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.855696 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gdff\" (UniqueName: \"kubernetes.io/projected/5e77fc9f-bada-453e-8b68-b57512b8850b-kube-api-access-8gdff\") pod \"payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:36.901780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:36.894166 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:37.270181 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:49:37.270148 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e77fc9f_bada_453e_8b68_b57512b8850b.slice/crio-710abdf0fb3b04cf8ace9b1adc882d1f4b0c553d7f45fdcc41326381b43f69c7 WatchSource:0}: Error finding container 710abdf0fb3b04cf8ace9b1adc882d1f4b0c553d7f45fdcc41326381b43f69c7: Status 404 returned error can't find the container with id 710abdf0fb3b04cf8ace9b1adc882d1f4b0c553d7f45fdcc41326381b43f69c7 Oct 30 03:49:37.270298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:37.270227 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529"] Oct 30 03:49:38.093095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.093049 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" podUID="5852cb33-795c-4f71-b43f-342ee3ad8e12" containerName="payload-processing" containerID="cri-o://5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126" gracePeriod=25 Oct 30 03:49:38.171968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.171933 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" event={"ID":"5e77fc9f-bada-453e-8b68-b57512b8850b","Type":"ContainerStarted","Data":"3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579"} Oct 30 03:49:38.172129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.171998 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" event={"ID":"5e77fc9f-bada-453e-8b68-b57512b8850b","Type":"ContainerStarted","Data":"710abdf0fb3b04cf8ace9b1adc882d1f4b0c553d7f45fdcc41326381b43f69c7"} Oct 30 03:49:38.172129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.172027 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:38.417962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.417937 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:49:38.434771 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.434727 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:38.459334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.459278 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" podStartSLOduration=2.459258889 podStartE2EDuration="2.459258889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:38.207899656 +0000 UTC m=+2802.210252853" watchObservedRunningTime="2026-10-30 03:49:38.459258889 +0000 UTC m=+2802.461612087" Oct 30 03:49:38.472647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.472620 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-routing-overlay-volume\") pod \"5852cb33-795c-4f71-b43f-342ee3ad8e12\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " Oct 30 03:49:38.472835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.472655 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-kube-api-access-8dxwg\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-kube-api-access-8dxwg\") pod \"5852cb33-795c-4f71-b43f-342ee3ad8e12\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " Oct 30 03:49:38.472835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.472721 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-provider-credentials\") pod \"5852cb33-795c-4f71-b43f-342ee3ad8e12\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " Oct 30 03:49:38.472994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.472893 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-plugins-config-volume\") pod \"5852cb33-795c-4f71-b43f-342ee3ad8e12\" (UID: \"5852cb33-795c-4f71-b43f-342ee3ad8e12\") " Oct 30 03:49:38.473341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.473310 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-routing-overlay-volume" pod "5852cb33-795c-4f71-b43f-342ee3ad8e12" (UID: "5852cb33-795c-4f71-b43f-342ee3ad8e12"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:38.473478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.473461 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:38.473711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.473682 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-plugins-config-volume" pod "5852cb33-795c-4f71-b43f-342ee3ad8e12" (UID: "5852cb33-795c-4f71-b43f-342ee3ad8e12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:38.475534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.475511 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-provider-credentials" pod "5852cb33-795c-4f71-b43f-342ee3ad8e12" (UID: "5852cb33-795c-4f71-b43f-342ee3ad8e12"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:49:38.478891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.478868 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-kube-api-access-8dxwg" pod "5852cb33-795c-4f71-b43f-342ee3ad8e12" (UID: "5852cb33-795c-4f71-b43f-342ee3ad8e12"). InnerVolumeSpecName "kube-api-access-8dxwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:49:38.598993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.591470 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5852cb33-795c-4f71-b43f-342ee3ad8e12-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:38.598993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.591509 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dxwg\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-kube-api-access-8dxwg\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:38.598993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:38.591524 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5852cb33-795c-4f71-b43f-342ee3ad8e12-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:39.181028 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.180996 4442 generic.go:356] "Generic (PLEG): container finished" podID="5852cb33-795c-4f71-b43f-342ee3ad8e12" containerID="5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126" exitCode=0 Oct 30 03:49:39.181526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.181051 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" Oct 30 03:49:39.181526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.181071 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" event={"ID":"5852cb33-795c-4f71-b43f-342ee3ad8e12","Type":"ContainerDied","Data":"5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126"} Oct 30 03:49:39.181526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.181120 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h" event={"ID":"5852cb33-795c-4f71-b43f-342ee3ad8e12","Type":"ContainerDied","Data":"2048fc49e5c21d3b4bb72a8ed7563fd7f04b08f278648746b60adcabd89a6485"} Oct 30 03:49:39.181526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.181140 4442 scope.go:122] "RemoveContainer" containerID="5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126" Oct 30 03:49:39.192064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.192050 4442 scope.go:122] "RemoveContainer" containerID="5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126" Oct 30 03:49:39.192335 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:49:39.192320 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126\": container with ID starting with 5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126 not found: ID does not exist" containerID="5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126" Oct 30 03:49:39.192376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.192342 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126"} err="failed to get container status \"5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126\": rpc error: code = NotFound desc = could not find container \"5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126\": container with ID starting with 5cb319b6a4895e3694ab0848770c87c6208098a8f8b4a4b718b50752a8165126 not found: ID does not exist" Oct 30 03:49:39.231868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.231842 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h"] Oct 30 03:49:39.241463 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.241438 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d4dzn6h"] Oct 30 03:49:39.929264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:39.929230 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:40.108849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:40.108814 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5852cb33-795c-4f71-b43f-342ee3ad8e12" path="/var/lib/kubelet/pods/5852cb33-795c-4f71-b43f-342ee3ad8e12/volumes" Oct 30 03:49:40.606520 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:40.606491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:40.708880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:40.708804 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:40.903456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:40.903363 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" podUID="4d20ac3c-12e5-4a6c-bb2a-e19807184269" containerName="payload-processing" containerID="cri-o://b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5" gracePeriod=25 Oct 30 03:49:41.180731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.180702 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:49:41.196647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.196618 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d20ac3c-12e5-4a6c-bb2a-e19807184269" containerID="b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5" exitCode=0 Oct 30 03:49:41.196801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.196667 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" Oct 30 03:49:41.196801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.196674 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" event={"ID":"4d20ac3c-12e5-4a6c-bb2a-e19807184269","Type":"ContainerDied","Data":"b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5"} Oct 30 03:49:41.196801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.196721 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg" event={"ID":"4d20ac3c-12e5-4a6c-bb2a-e19807184269","Type":"ContainerDied","Data":"983d976ae89b04bd49fefd091fe31d38f47105499e06bb05882aab776c7f4ff3"} Oct 30 03:49:41.196801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.196742 4442 scope.go:122] "RemoveContainer" containerID="b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5" Oct 30 03:49:41.206824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.206804 4442 scope.go:122] "RemoveContainer" containerID="b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5" Oct 30 03:49:41.207072 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:49:41.207054 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5\": container with ID starting with b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5 not found: ID does not exist" containerID="b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5" Oct 30 03:49:41.207133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.207082 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5"} err="failed to get container status \"b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5\": rpc error: code = NotFound desc = could not find container \"b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5\": container with ID starting with b5faa6eea6d609f845de8482db6e66dba918135d161d6e86a1d73071133f25e5 not found: ID does not exist" Oct 30 03:49:41.238198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.238171 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-plugins-config-volume\") pod \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " Oct 30 03:49:41.238329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.238221 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-provider-credentials\") pod \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " Oct 30 03:49:41.238329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.238293 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-routing-overlay-volume\") pod \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " Oct 30 03:49:41.238505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.238485 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-kube-api-access-4fvpk\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-kube-api-access-4fvpk\") pod \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\" (UID: \"4d20ac3c-12e5-4a6c-bb2a-e19807184269\") " Oct 30 03:49:41.238740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.238708 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-plugins-config-volume" pod "4d20ac3c-12e5-4a6c-bb2a-e19807184269" (UID: "4d20ac3c-12e5-4a6c-bb2a-e19807184269"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:41.238878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.238774 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-routing-overlay-volume" pod "4d20ac3c-12e5-4a6c-bb2a-e19807184269" (UID: "4d20ac3c-12e5-4a6c-bb2a-e19807184269"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:49:41.239386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.239353 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:41.239386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.239382 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4d20ac3c-12e5-4a6c-bb2a-e19807184269-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:41.240796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.240737 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-provider-credentials" pod "4d20ac3c-12e5-4a6c-bb2a-e19807184269" (UID: "4d20ac3c-12e5-4a6c-bb2a-e19807184269"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:49:41.241451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.241419 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-kube-api-access-4fvpk" pod "4d20ac3c-12e5-4a6c-bb2a-e19807184269" (UID: "4d20ac3c-12e5-4a6c-bb2a-e19807184269"). InnerVolumeSpecName "kube-api-access-4fvpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:49:41.340480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.340450 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fvpk\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-kube-api-access-4fvpk\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:41.340480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.340480 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4d20ac3c-12e5-4a6c-bb2a-e19807184269-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:49:41.540928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.540895 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg"] Oct 30 03:49:41.552732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.552694 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99q9zwg"] Oct 30 03:49:41.599159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.599116 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:41.699367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:41.699331 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:42.129594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:42.122225 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d20ac3c-12e5-4a6c-bb2a-e19807184269" path="/var/lib/kubelet/pods/4d20ac3c-12e5-4a6c-bb2a-e19807184269/volumes" Oct 30 03:49:43.083692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.083655 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:43.227696 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.227663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:43.355532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.355456 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-66f6857b97-np5bw"] Oct 30 03:49:43.356169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.356155 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5852cb33-795c-4f71-b43f-342ee3ad8e12" containerName="payload-processing" Oct 30 03:49:43.356230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.356171 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5852cb33-795c-4f71-b43f-342ee3ad8e12" containerName="payload-processing" Oct 30 03:49:43.356230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.356190 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d20ac3c-12e5-4a6c-bb2a-e19807184269" containerName="payload-processing" Oct 30 03:49:43.356230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.356195 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d20ac3c-12e5-4a6c-bb2a-e19807184269" containerName="payload-processing" Oct 30 03:49:43.356327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.356308 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5852cb33-795c-4f71-b43f-342ee3ad8e12" containerName="payload-processing" Oct 30 03:49:43.356327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.356317 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d20ac3c-12e5-4a6c-bb2a-e19807184269" containerName="payload-processing" Oct 30 03:49:43.361221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.361201 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.378285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.378261 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f6857b97-np5bw"] Oct 30 03:49:43.384559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.384532 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e9fcc7de5ef8\"" Oct 30 03:49:43.384695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.384632 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 03:49:43.384695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.384659 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 03:49:43.384695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.384691 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 03:49:43.386930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.386910 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 03:49:43.387034 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.386962 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-l6dcw\"" Oct 30 03:49:43.482822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.481818 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-routing-overlay-volume\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.490910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.490870 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjksj\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-kube-api-access-zjksj\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.491314 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.491289 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-provider-credentials\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.511007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.510966 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-plugins-config-volume\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.612836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.612668 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-routing-overlay-volume\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.612836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.612730 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjksj\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-kube-api-access-zjksj\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.612836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.612822 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-provider-credentials\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.613167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.612909 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-plugins-config-volume\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.620222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.614203 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-plugins-config-volume\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.621542 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.621479 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-routing-overlay-volume\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.628530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.628506 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-provider-credentials\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.642513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.642458 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjksj\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-kube-api-access-zjksj\") pod \"payload-processing-external-model-66f6857b97-np5bw\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:43.679428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:43.677938 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:44.022679 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:49:44.022638 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05478738_d404_4226_b746_040cf22dc02d.slice/crio-dc2c0232ce4157f2042c7013da8d6d4c4626eec54091697118e75be520168607 WatchSource:0}: Error finding container dc2c0232ce4157f2042c7013da8d6d4c4626eec54091697118e75be520168607: Status 404 returned error can't find the container with id dc2c0232ce4157f2042c7013da8d6d4c4626eec54091697118e75be520168607 Oct 30 03:49:44.025177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:44.025149 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f6857b97-np5bw"] Oct 30 03:49:44.220416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:44.220375 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" event={"ID":"05478738-d404-4226-b746-040cf22dc02d","Type":"ContainerStarted","Data":"092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc"} Oct 30 03:49:44.220831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:44.220436 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" event={"ID":"05478738-d404-4226-b746-040cf22dc02d","Type":"ContainerStarted","Data":"dc2c0232ce4157f2042c7013da8d6d4c4626eec54091697118e75be520168607"} Oct 30 03:49:44.220831 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:44.220608 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:44.255814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:44.255766 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" podStartSLOduration=1.255736599 podStartE2EDuration="1.255736599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:49:44.255464013 +0000 UTC m=+2808.257817210" watchObservedRunningTime="2026-10-30 03:49:44.255736599 +0000 UTC m=+2808.258089796" Oct 30 03:49:44.809276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:44.809247 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:46.373903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:46.373869 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:47.890131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:47.890098 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:49.185035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:49.185006 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:49:49.385924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:49.385893 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:50.058352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:50.058317 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:51.004713 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:51.004678 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:51.298025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:51.297947 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:52.815089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:52.815056 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:53.038319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:53.038291 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:54.368009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:54.367977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:54.862526 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:49:54.862480 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:49:55.231516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:55.231433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:49:56.105488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:56.105446 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:57.789744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:57.789707 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:49:59.355193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:49:59.355160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:00.824225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:00.824194 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:02.401881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:02.401840 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:03.913530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:03.913499 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:03.996879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:03.996850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:05.802300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:05.802268 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:07.503602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:07.503570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:08.676487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:08.676447 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:50:09.080627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:09.080592 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:10.628236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:10.628196 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:11.147142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:11.147090 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:13.422559 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:13.422530 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:13.531780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:13.531074 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:13.622469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:13.622438 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:15.222364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:15.222324 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:16.818459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:16.818429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:18.439977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:18.439942 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:19.778854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:19.778824 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:19.881092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:19.881058 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:20.050832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.040604 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62"] Oct 30 03:50:20.167987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.167945 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:20.198000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.197967 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.216573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.216552 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62"] Oct 30 03:50:20.218849 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.217637 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-35fc87-serving-cert\"" Oct 30 03:50:20.278536 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.278501 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:20.288576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.288541 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dng\" (UniqueName: \"kubernetes.io/projected/f2c55a01-06ca-407d-9f40-bcd70007c00a-kube-api-access-n2dng\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.289024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.288998 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-maas-api-tls\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.289167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.289151 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-metrics-certs\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.289356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.289337 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c55a01-06ca-407d-9f40-bcd70007c00a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.373518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.373420 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:20.409816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.405716 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-maas-api-tls\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.409816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.405795 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-metrics-certs\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.409816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.405876 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c55a01-06ca-407d-9f40-bcd70007c00a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.409816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.406096 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dng\" (UniqueName: \"kubernetes.io/projected/f2c55a01-06ca-407d-9f40-bcd70007c00a-kube-api-access-n2dng\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.418609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.418573 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c55a01-06ca-407d-9f40-bcd70007c00a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.433873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.433835 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-metrics-certs\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.448541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.448470 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dng\" (UniqueName: \"kubernetes.io/projected/f2c55a01-06ca-407d-9f40-bcd70007c00a-kube-api-access-n2dng\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.460857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.460824 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-maas-api-tls\") pod \"maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:20.530344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:20.530299 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:21.014205 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.012099 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62"] Oct 30 03:50:21.033817 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:50:21.033712 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c55a01_06ca_407d_9f40_bcd70007c00a.slice/crio-ef48ab90db94bd8cd69b60503cbbb97171e9f1e1cee3eb78dd92a0dff0c25f8a WatchSource:0}: Error finding container ef48ab90db94bd8cd69b60503cbbb97171e9f1e1cee3eb78dd92a0dff0c25f8a: Status 404 returned error can't find the container with id ef48ab90db94bd8cd69b60503cbbb97171e9f1e1cee3eb78dd92a0dff0c25f8a Oct 30 03:50:21.043017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.041799 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:50:21.452279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.452246 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p"] Oct 30 03:50:21.457247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.457223 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.480731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.480701 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p"] Oct 30 03:50:21.486179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.485490 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-plugins-e2e-extmod-a-e10c0f\"" Oct 30 03:50:21.486179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.485613 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"routing-overlay\"" Oct 30 03:50:21.486179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.485794 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"payload-processing-external-model-e2e--4f8d4144-dockercfg-wmn86\"" Oct 30 03:50:21.486179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.485873 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"agc-cred-681a94cb0ee9\"" Oct 30 03:50:21.486179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.485935 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"openshift-service-ca.crt\"" Oct 30 03:50:21.486179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.486034 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"kube-root-ca.crt\"" Oct 30 03:50:21.486179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.486073 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"agc-cred-eda87f61d8ec\"" Oct 30 03:50:21.489337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.486334 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-e10c0f\"/\"agc-cred-5cfdc2e397bd\"" Oct 30 03:50:21.542908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.542876 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.543082 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.542929 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.543161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.543136 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.543212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.543201 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9mms\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-kube-api-access-l9mms\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.644691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.644599 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.644894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.644695 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9mms\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-kube-api-access-l9mms\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.644894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.644795 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.644894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.644829 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.645563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.645529 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.645684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.645532 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.648077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.648053 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.689786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.674155 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9mms\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-kube-api-access-l9mms\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.778620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.778203 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:21.793420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.790145 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" event={"ID":"f2c55a01-06ca-407d-9f40-bcd70007c00a","Type":"ContainerStarted","Data":"fa287b11dc833270445ec787ade2d2a3a695513ca56574bd8167e82312b5f4ed"} Oct 30 03:50:21.793420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.790337 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" event={"ID":"f2c55a01-06ca-407d-9f40-bcd70007c00a","Type":"ContainerStarted","Data":"ef48ab90db94bd8cd69b60503cbbb97171e9f1e1cee3eb78dd92a0dff0c25f8a"} Oct 30 03:50:21.794837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.794267 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:21.838462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:21.838397 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" podStartSLOduration=2.5496097 podStartE2EDuration="2.838373905s" totalImagesPullingTime="288.764205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:21.835129577 +0000 UTC m=+2845.837482774" watchObservedRunningTime="2026-10-30 03:50:21.838373905 +0000 UTC m=+2845.840727103" Oct 30 03:50:22.108067 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:50:22.108031 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ac42f3_5809_466a_bd91_fed7fc181ffc.slice/crio-b8eada1d297b8427952237d3ed532f70a0445f5f6ecbd1517940bb978fdbffbb WatchSource:0}: Error finding container b8eada1d297b8427952237d3ed532f70a0445f5f6ecbd1517940bb978fdbffbb: Status 404 returned error can't find the container with id b8eada1d297b8427952237d3ed532f70a0445f5f6ecbd1517940bb978fdbffbb Oct 30 03:50:22.109607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.109583 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p"] Oct 30 03:50:22.425603 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.425519 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:22.442784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.442479 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx"] Oct 30 03:50:22.447167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.447143 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.465167 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.465139 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-372480\"/\"agc-cred-0b02f17d5fce\"" Oct 30 03:50:22.532539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.532119 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx"] Oct 30 03:50:22.563888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.563848 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.564082 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.563924 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.564142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.564120 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9n7q\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-kube-api-access-d9n7q\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.564184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.564154 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.665190 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.665158 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.665354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.665203 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.665354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.665292 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9n7q\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-kube-api-access-d9n7q\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.665354 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.665310 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.665947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.665918 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.666076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.666058 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.668052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.668028 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.684597 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.684537 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9n7q\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-kube-api-access-d9n7q\") pod \"payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.762474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.762447 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:22.800860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.800822 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" event={"ID":"a2ac42f3-5809-466a-bd91-fed7fc181ffc","Type":"ContainerStarted","Data":"806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053"} Oct 30 03:50:22.801035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.800891 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" event={"ID":"a2ac42f3-5809-466a-bd91-fed7fc181ffc","Type":"ContainerStarted","Data":"b8eada1d297b8427952237d3ed532f70a0445f5f6ecbd1517940bb978fdbffbb"} Oct 30 03:50:22.801035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.800966 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:22.843380 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:22.843322 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" podStartSLOduration=1.843307556 podStartE2EDuration="1.843307556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:22.839258414 +0000 UTC m=+2846.841611611" watchObservedRunningTime="2026-10-30 03:50:22.843307556 +0000 UTC m=+2846.845660753" Oct 30 03:50:23.068928 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:50:23.068887 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78456d8_5a07_4d22_9e43_0c034a0cd729.slice/crio-a0f035cbabb900e526c584cc3c6af793cc4ad383bcbde7ff7e0a852f0c8ad55d WatchSource:0}: Error finding container a0f035cbabb900e526c584cc3c6af793cc4ad383bcbde7ff7e0a852f0c8ad55d: Status 404 returned error can't find the container with id a0f035cbabb900e526c584cc3c6af793cc4ad383bcbde7ff7e0a852f0c8ad55d Oct 30 03:50:23.068928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:23.068900 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx"] Oct 30 03:50:23.809181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:23.809132 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" event={"ID":"c78456d8-5a07-4d22-9e43-0c034a0cd729","Type":"ContainerStarted","Data":"9fd93c3d7826e5b71dfe33ee2c785ab351875a29d6e94ee392656df60319b8e0"} Oct 30 03:50:23.810153 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:23.809304 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:23.810153 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:23.809362 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" event={"ID":"c78456d8-5a07-4d22-9e43-0c034a0cd729","Type":"ContainerStarted","Data":"a0f035cbabb900e526c584cc3c6af793cc4ad383bcbde7ff7e0a852f0c8ad55d"} Oct 30 03:50:23.849576 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:23.849528 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" podStartSLOduration=1.849514309 podStartE2EDuration="1.849514309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:23.847713688 +0000 UTC m=+2847.850066886" watchObservedRunningTime="2026-10-30 03:50:23.849514309 +0000 UTC m=+2847.851867505" Oct 30 03:50:24.013836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:24.013804 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:24.863228 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:50:24.863188 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:50:25.106003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:25.105960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:25.204285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:25.204201 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:25.774653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:25.774613 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f6857b97-np5bw"] Oct 30 03:50:25.843037 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:25.843007 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:25.942654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:25.942611 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:27.215612 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:27.215581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:27.557297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:27.557241 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:27.679976 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:27.679942 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:27.806298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:27.806246 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:28.815491 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:28.815464 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:50:29.205278 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:29.205193 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:30.776127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:30.776088 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" podUID="05478738-d404-4226-b746-040cf22dc02d" containerName="payload-processing" containerID="cri-o://092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc" gracePeriod=25 Oct 30 03:50:30.825618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:30.825589 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:31.117766 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.117728 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:50:31.134788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.134748 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-kube-api-access-zjksj\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-kube-api-access-zjksj\") pod \"05478738-d404-4226-b746-040cf22dc02d\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " Oct 30 03:50:31.134947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.134815 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-routing-overlay-volume\") pod \"05478738-d404-4226-b746-040cf22dc02d\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " Oct 30 03:50:31.134947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.134895 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-provider-credentials\") pod \"05478738-d404-4226-b746-040cf22dc02d\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " Oct 30 03:50:31.134947 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.134940 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-plugins-config-volume\") pod \"05478738-d404-4226-b746-040cf22dc02d\" (UID: \"05478738-d404-4226-b746-040cf22dc02d\") " Oct 30 03:50:31.135335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.135295 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-routing-overlay-volume" pod "05478738-d404-4226-b746-040cf22dc02d" (UID: "05478738-d404-4226-b746-040cf22dc02d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:31.135445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.135351 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-plugins-config-volume" pod "05478738-d404-4226-b746-040cf22dc02d" (UID: "05478738-d404-4226-b746-040cf22dc02d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:31.135549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.135533 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:31.135589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.135552 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05478738-d404-4226-b746-040cf22dc02d-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:31.137744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.137723 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-provider-credentials" pod "05478738-d404-4226-b746-040cf22dc02d" (UID: "05478738-d404-4226-b746-040cf22dc02d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:31.137851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.137737 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-kube-api-access-zjksj" pod "05478738-d404-4226-b746-040cf22dc02d" (UID: "05478738-d404-4226-b746-040cf22dc02d"). InnerVolumeSpecName "kube-api-access-zjksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:31.237310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.237276 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjksj\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-kube-api-access-zjksj\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:31.237310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.237304 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/05478738-d404-4226-b746-040cf22dc02d-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:31.600507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.600475 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:31.706368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.706334 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:31.948114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.948041 4442 generic.go:356] "Generic (PLEG): container finished" podID="05478738-d404-4226-b746-040cf22dc02d" containerID="092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc" exitCode=0 Oct 30 03:50:31.948114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.948085 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" Oct 30 03:50:31.948639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.948123 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" event={"ID":"05478738-d404-4226-b746-040cf22dc02d","Type":"ContainerDied","Data":"092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc"} Oct 30 03:50:31.948639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.948190 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-66f6857b97-np5bw" event={"ID":"05478738-d404-4226-b746-040cf22dc02d","Type":"ContainerDied","Data":"dc2c0232ce4157f2042c7013da8d6d4c4626eec54091697118e75be520168607"} Oct 30 03:50:31.948639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.948212 4442 scope.go:122] "RemoveContainer" containerID="092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc" Oct 30 03:50:31.960833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.960811 4442 scope.go:122] "RemoveContainer" containerID="092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc" Oct 30 03:50:31.961115 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:50:31.961092 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc\": container with ID starting with 092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc not found: ID does not exist" containerID="092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc" Oct 30 03:50:31.961161 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:31.961128 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc"} err="failed to get container status \"092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc\": rpc error: code = NotFound desc = could not find container \"092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc\": container with ID starting with 092dccf210612a375f0d551431a1d72cfeaed44d90a204ce8457533542fdbcfc not found: ID does not exist" Oct 30 03:50:32.020684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:32.020646 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f6857b97-np5bw"] Oct 30 03:50:32.040916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:32.040890 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-66f6857b97-np5bw"] Oct 30 03:50:32.108863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:32.108836 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05478738-d404-4226-b746-040cf22dc02d" path="/var/lib/kubelet/pods/05478738-d404-4226-b746-040cf22dc02d/volumes" Oct 30 03:50:32.697291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:32.697259 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:33.397501 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.397467 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:33.497530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.497494 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:33.534087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.534053 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c"] Oct 30 03:50:33.534784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.534770 4442 cpu_manager.go:414] "containerMap: removing container" podUID="05478738-d404-4226-b746-040cf22dc02d" containerName="payload-processing" Oct 30 03:50:33.534863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.534785 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05478738-d404-4226-b746-040cf22dc02d" containerName="payload-processing" Oct 30 03:50:33.534935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.534925 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="05478738-d404-4226-b746-040cf22dc02d" containerName="payload-processing" Oct 30 03:50:33.539785 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.539745 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.561133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.561105 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c"] Oct 30 03:50:33.566922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.566899 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3cfed7\"" Oct 30 03:50:33.575695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.575651 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjj2\" (UniqueName: \"kubernetes.io/projected/abf331bc-81be-422e-bea5-34b16157cb52-kube-api-access-hqjj2\") pod \"payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.575859 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.575723 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf331bc-81be-422e-bea5-34b16157cb52-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.604926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.604896 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf"] Oct 30 03:50:33.609457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.609440 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:33.623442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.623392 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3cfed7-dockercfg-kxv5h\"" Oct 30 03:50:33.627925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.627901 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf"] Oct 30 03:50:33.713832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.703860 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjj2\" (UniqueName: \"kubernetes.io/projected/abf331bc-81be-422e-bea5-34b16157cb52-kube-api-access-hqjj2\") pod \"payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.713832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.703937 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf331bc-81be-422e-bea5-34b16157cb52-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.713832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.703982 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d989d15-7886-4cac-b022-13d584a23039-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:33.738776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.738703 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf331bc-81be-422e-bea5-34b16157cb52-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.777198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.774296 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwqr\" (UniqueName: \"kubernetes.io/projected/9d989d15-7886-4cac-b022-13d584a23039-kube-api-access-grwqr\") pod \"payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:33.858781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.853916 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjj2\" (UniqueName: \"kubernetes.io/projected/abf331bc-81be-422e-bea5-34b16157cb52-kube-api-access-hqjj2\") pod \"payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.858781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.855150 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:33.882812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.875778 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwqr\" (UniqueName: \"kubernetes.io/projected/9d989d15-7886-4cac-b022-13d584a23039-kube-api-access-grwqr\") pod \"payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:33.882812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.875919 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d989d15-7886-4cac-b022-13d584a23039-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:33.882812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.877135 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d989d15-7886-4cac-b022-13d584a23039-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:33.882812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.878639 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:50:33.900773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.900376 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwqr\" (UniqueName: \"kubernetes.io/projected/9d989d15-7886-4cac-b022-13d584a23039-kube-api-access-grwqr\") pod \"payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:33.923645 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:33.923614 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:34.244849 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:50:34.244814 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf331bc_81be_422e_bea5_34b16157cb52.slice/crio-28f1b3e575be545aaef0e982e0572f81c629d3fe368c95481de9c8ef6378e448 WatchSource:0}: Error finding container 28f1b3e575be545aaef0e982e0572f81c629d3fe368c95481de9c8ef6378e448: Status 404 returned error can't find the container with id 28f1b3e575be545aaef0e982e0572f81c629d3fe368c95481de9c8ef6378e448 Oct 30 03:50:34.246648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.246615 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c"] Oct 30 03:50:34.270377 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.270350 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf"] Oct 30 03:50:34.272241 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:50:34.272213 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d989d15_7886_4cac_b022_13d584a23039.slice/crio-99527bb53294cc4304a6a84a6a988d73e860af96948865c26f53096012e727af WatchSource:0}: Error finding container 99527bb53294cc4304a6a84a6a988d73e860af96948865c26f53096012e727af: Status 404 returned error can't find the container with id 99527bb53294cc4304a6a84a6a988d73e860af96948865c26f53096012e727af Oct 30 03:50:34.514631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.514599 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:34.820336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.820252 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:50:34.890065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.890032 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn"] Oct 30 03:50:34.993035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.992999 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" event={"ID":"abf331bc-81be-422e-bea5-34b16157cb52","Type":"ContainerStarted","Data":"48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775"} Oct 30 03:50:34.993237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.993073 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:34.993237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.993101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" event={"ID":"abf331bc-81be-422e-bea5-34b16157cb52","Type":"ContainerStarted","Data":"28f1b3e575be545aaef0e982e0572f81c629d3fe368c95481de9c8ef6378e448"} Oct 30 03:50:34.994727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.994701 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" event={"ID":"9d989d15-7886-4cac-b022-13d584a23039","Type":"ContainerStarted","Data":"1e0d743525c5496e6f84b7a7a04f116962fa87ace29c92b98ae677543329de29"} Oct 30 03:50:34.994899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.994747 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" event={"ID":"9d989d15-7886-4cac-b022-13d584a23039","Type":"ContainerStarted","Data":"99527bb53294cc4304a6a84a6a988d73e860af96948865c26f53096012e727af"} Oct 30 03:50:34.994899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:34.994819 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:35.036027 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:35.035978 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" podStartSLOduration=2.03596385 podStartE2EDuration="2.03596385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:35.032725902 +0000 UTC m=+2859.035079100" watchObservedRunningTime="2026-10-30 03:50:35.03596385 +0000 UTC m=+2859.038317047" Oct 30 03:50:35.076365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:35.076265 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" podStartSLOduration=2.076252829 podStartE2EDuration="2.076252829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:35.073924552 +0000 UTC m=+2859.076277769" watchObservedRunningTime="2026-10-30 03:50:35.076252829 +0000 UTC m=+2859.078606025" Oct 30 03:50:36.190720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:36.190684 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:36.692914 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:36.692879 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:37.718457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:37.718414 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:39.424990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:39.424943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:39.890741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:39.890690 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" podUID="79d8dda4-21d6-4348-97db-8934f196dbf8" containerName="payload-processing" containerID="cri-o://34be0cad90eb09d419713c66a29bc0762c3ac6a308e6d0cac1abb3b7e0817c11" gracePeriod=25 Oct 30 03:50:40.045808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.045770 4442 generic.go:356] "Generic (PLEG): container finished" podID="79d8dda4-21d6-4348-97db-8934f196dbf8" containerID="34be0cad90eb09d419713c66a29bc0762c3ac6a308e6d0cac1abb3b7e0817c11" exitCode=0 Oct 30 03:50:40.045808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.045803 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" event={"ID":"79d8dda4-21d6-4348-97db-8934f196dbf8","Type":"ContainerDied","Data":"34be0cad90eb09d419713c66a29bc0762c3ac6a308e6d0cac1abb3b7e0817c11"} Oct 30 03:50:40.165579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.165555 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:50:40.273251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.273222 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-provider-credentials\") pod \"79d8dda4-21d6-4348-97db-8934f196dbf8\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " Oct 30 03:50:40.273428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.273328 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-kube-api-access-bxxmc\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-kube-api-access-bxxmc\") pod \"79d8dda4-21d6-4348-97db-8934f196dbf8\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " Oct 30 03:50:40.273516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.273453 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-routing-overlay-volume\") pod \"79d8dda4-21d6-4348-97db-8934f196dbf8\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " Oct 30 03:50:40.273516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.273485 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-plugins-config-volume\") pod \"79d8dda4-21d6-4348-97db-8934f196dbf8\" (UID: \"79d8dda4-21d6-4348-97db-8934f196dbf8\") " Oct 30 03:50:40.273863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.273832 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-routing-overlay-volume" pod "79d8dda4-21d6-4348-97db-8934f196dbf8" (UID: "79d8dda4-21d6-4348-97db-8934f196dbf8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:40.273952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.273924 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-plugins-config-volume" pod "79d8dda4-21d6-4348-97db-8934f196dbf8" (UID: "79d8dda4-21d6-4348-97db-8934f196dbf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:50:40.274181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.274067 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:40.274181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.274081 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79d8dda4-21d6-4348-97db-8934f196dbf8-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:40.275589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.275563 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-kube-api-access-bxxmc" pod "79d8dda4-21d6-4348-97db-8934f196dbf8" (UID: "79d8dda4-21d6-4348-97db-8934f196dbf8"). InnerVolumeSpecName "kube-api-access-bxxmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:40.275589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.275581 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-provider-credentials" pod "79d8dda4-21d6-4348-97db-8934f196dbf8" (UID: "79d8dda4-21d6-4348-97db-8934f196dbf8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:50:40.375226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.375194 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxxmc\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-kube-api-access-bxxmc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:40.375226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:40.375224 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79d8dda4-21d6-4348-97db-8934f196dbf8-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:50:41.054347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:41.054321 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" Oct 30 03:50:41.054818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:41.054313 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn" event={"ID":"79d8dda4-21d6-4348-97db-8934f196dbf8","Type":"ContainerDied","Data":"e2823b9b97f20c1aaae10e13a667b38f6506a93f50ac30f9507c7e2171c6971a"} Oct 30 03:50:41.054818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:41.054474 4442 scope.go:122] "RemoveContainer" containerID="34be0cad90eb09d419713c66a29bc0762c3ac6a308e6d0cac1abb3b7e0817c11" Oct 30 03:50:41.110857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:41.110830 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn"] Oct 30 03:50:41.121514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:41.121493 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-6db98stcn"] Oct 30 03:50:41.125794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:41.125775 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:42.117358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:42.117323 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d8dda4-21d6-4348-97db-8934f196dbf8" path="/var/lib/kubelet/pods/79d8dda4-21d6-4348-97db-8934f196dbf8/volumes" Oct 30 03:50:42.780382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:42.780337 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:44.994875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:44.994841 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:46.006528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:46.006497 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:50:46.006974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:46.006614 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:50:46.917178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:46.917148 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:48.689410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:48.689371 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:50.253958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:50.253924 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:50.330574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:50.330543 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:51.151498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:51.151449 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:51.772967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:51.772929 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:53.183726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:53.183647 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:53.412655 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:53.412623 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:54.945098 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:50:54.945061 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:50:55.002460 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:55.002429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:56.620010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:56.619972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:58.327394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:58.327350 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:58.420202 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:58.420038 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:59.769701 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:59.769659 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:50:59.945788 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:50:59.945725 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:01.089932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.089898 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms"] Oct 30 03:51:01.090736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.090719 4442 cpu_manager.go:414] "containerMap: removing container" podUID="79d8dda4-21d6-4348-97db-8934f196dbf8" containerName="payload-processing" Oct 30 03:51:01.090825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.090737 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d8dda4-21d6-4348-97db-8934f196dbf8" containerName="payload-processing" Oct 30 03:51:01.090914 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.090903 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d8dda4-21d6-4348-97db-8934f196dbf8" containerName="payload-processing" Oct 30 03:51:01.094653 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.094638 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.112283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.112260 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-35fc87-openshift-default-dockercfg-n4t6x\"" Oct 30 03:51:01.128210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.128181 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms"] Oct 30 03:51:01.173365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.173312 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.173556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.173382 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.173556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.173515 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.173778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.173647 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.173866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.173809 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.174047 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.174023 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac681208-ea26-4098-ada6-0315accf7fcf-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.174146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.174074 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.174146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.174122 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac681208-ea26-4098-ada6-0315accf7fcf-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.174269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.174193 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jr6\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-kube-api-access-t4jr6\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275381 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275435 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275488 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275581 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275685 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275781 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac681208-ea26-4098-ada6-0315accf7fcf-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275817 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.275860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275842 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac681208-ea26-4098-ada6-0315accf7fcf-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.276067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275872 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jr6\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-kube-api-access-t4jr6\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.276067 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.275888 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.276165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.276100 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.276342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.276316 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.276464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.276399 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.276535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.276515 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac681208-ea26-4098-ada6-0315accf7fcf-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.278060 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.278041 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.278503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.278481 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac681208-ea26-4098-ada6-0315accf7fcf-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.294352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.294329 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.294497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.294475 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jr6\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-kube-api-access-t4jr6\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.409550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.409472 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:01.690618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.690535 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:01.746879 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.742816 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms"] Oct 30 03:51:01.747357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.747185 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:01.747357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.747295 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:01.747357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.747355 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:01.874940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.874905 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp"] Oct 30 03:51:01.881647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.881629 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.899234 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.899205 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e84013-openshift-default-dockercfg-4sn84\"" Oct 30 03:51:01.904024 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.903988 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp"] Oct 30 03:51:01.992512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.992413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.992512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.992465 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.992776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.992617 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq5vr\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-kube-api-access-vq5vr\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.992776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.992715 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.992872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.992806 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.993005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.992982 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.993095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.993078 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.993150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.993135 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:01.993197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:01.993169 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.094946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.094904 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095052 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095092 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095143 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095273 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095315 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095386 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq5vr\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-kube-api-access-vq5vr\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095448 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095513 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.095943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.095852 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.096172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.096149 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.096442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.096364 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.096442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.096432 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.098843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.098820 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.099705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.099666 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.117342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.117306 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.117495 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.117422 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq5vr\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-kube-api-access-vq5vr\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.197734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.197699 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:02.242106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.240388 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" event={"ID":"ac681208-ea26-4098-ada6-0315accf7fcf","Type":"ContainerStarted","Data":"cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc"} Oct 30 03:51:02.242106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.240458 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" event={"ID":"ac681208-ea26-4098-ada6-0315accf7fcf","Type":"ContainerStarted","Data":"962fdf180cc471fc09f290a3c293bf1c586b35650a3a712bf76bc5f680c7cc95"} Oct 30 03:51:02.289642 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.289421 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podStartSLOduration=1.28940221 podStartE2EDuration="1.28940221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:02.28719633 +0000 UTC m=+2886.289549539" watchObservedRunningTime="2026-10-30 03:51:02.28940221 +0000 UTC m=+2886.291755408" Oct 30 03:51:02.409899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.409867 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:02.411268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.411246 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 03:51:02.411379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.411298 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 03:51:02.539102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.538350 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp"] Oct 30 03:51:02.542429 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:51:02.542404 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f78fc12_1193_44e0_ad47_d0db6a795b3a.slice/crio-14af6dab89686b809bfa2ebc63c4e9356392cc59c96a64a64e9e08fc92d92e23 WatchSource:0}: Error finding container 14af6dab89686b809bfa2ebc63c4e9356392cc59c96a64a64e9e08fc92d92e23: Status 404 returned error can't find the container with id 14af6dab89686b809bfa2ebc63c4e9356392cc59c96a64a64e9e08fc92d92e23 Oct 30 03:51:02.545376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.545345 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:02.545453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.545415 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:02.545453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:02.545443 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:03.254151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:03.254101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" event={"ID":"9f78fc12-1193-44e0-ad47-d0db6a795b3a","Type":"ContainerStarted","Data":"c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a"} Oct 30 03:51:03.254638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:03.254175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" event={"ID":"9f78fc12-1193-44e0-ad47-d0db6a795b3a","Type":"ContainerStarted","Data":"14af6dab89686b809bfa2ebc63c4e9356392cc59c96a64a64e9e08fc92d92e23"} Oct 30 03:51:03.319880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:03.317405 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" podStartSLOduration=2.317380653 podStartE2EDuration="2.317380653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:03.305498301 +0000 UTC m=+2887.307851520" watchObservedRunningTime="2026-10-30 03:51:03.317380653 +0000 UTC m=+2887.319733851" Oct 30 03:51:03.322018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:03.321981 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:03.424780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:03.417279 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 03:51:03.424780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:03.417343 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 03:51:04.207480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:04.207400 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:04.223779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:04.210036 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 03:51:04.223779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:04.210117 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 03:51:04.410778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:04.410696 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 03:51:04.411351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:04.410774 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 03:51:05.198778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:05.198134 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 03:51:05.198778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:05.198235 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 03:51:05.306778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:05.306651 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:05.443482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:05.443443 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 03:51:05.444118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:05.444083 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 03:51:06.211780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:06.205843 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 03:51:06.211780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:06.205908 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 03:51:06.414780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:06.411622 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 03:51:06.414780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:06.411690 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 03:51:07.199780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:07.198411 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 03:51:07.199780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:07.198464 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 03:51:07.240680 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:07.240653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:07.414243 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:07.414201 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 03:51:07.414555 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:07.414521 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 03:51:08.216884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:08.214408 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 30 03:51:08.217689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:08.217642 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 30 03:51:08.448277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:08.448237 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:08.449365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:08.449342 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:08.450112 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:08.450084 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:51:08.996575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:08.996514 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:09.204196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:09.204170 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:09.204665 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:09.204647 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:09.205306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:09.205291 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:51:10.809227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:10.809192 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:11.997285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:11.997215 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:12.460498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:12.460461 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:12.594783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:12.572743 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:12.869203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:12.869155 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:12.949981 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:12.949937 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4"] Oct 30 03:51:12.978800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:12.976316 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:12.987819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:12.987659 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4"] Oct 30 03:51:13.002782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.001746 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.002782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.001894 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdxhr\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-kube-api-access-mdxhr\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.002782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.001969 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.002782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.002065 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.103072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.103035 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.103239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.103198 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.103285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.103239 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.103333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.103290 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdxhr\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-kube-api-access-mdxhr\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.103975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.103946 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.104058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.103996 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.105830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.105800 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.130406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.130345 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdxhr\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-kube-api-access-mdxhr\") pod \"payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.438106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.438014 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:13.795959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:13.795927 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4"] Oct 30 03:51:13.796289 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:51:13.796262 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66514a82_468a_4b6f_a5fe_08ac2470d27e.slice/crio-b9bf4fe86dddfe64f99829a6031a196cf9cb28f9427d3928666ad48720c6f529 WatchSource:0}: Error finding container b9bf4fe86dddfe64f99829a6031a196cf9cb28f9427d3928666ad48720c6f529: Status 404 returned error can't find the container with id b9bf4fe86dddfe64f99829a6031a196cf9cb28f9427d3928666ad48720c6f529 Oct 30 03:51:14.190091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.189994 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh"] Oct 30 03:51:14.197793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.197092 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.220498 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.220470 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh"] Oct 30 03:51:14.222320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.222299 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"agc-cred-7b4014be3195\"" Oct 30 03:51:14.222594 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.222577 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"openshift-service-ca.crt\"" Oct 30 03:51:14.222935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.222918 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-e2e--22d06fb9-dockercfg-kmzx9\"" Oct 30 03:51:14.223138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.223120 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"kube-root-ca.crt\"" Oct 30 03:51:14.231919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.231899 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"routing-overlay\"" Oct 30 03:51:14.232026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.231961 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-689bcf\"/\"payload-processing-external-model-plugins-e2e-extmod-a-689bcf\"" Oct 30 03:51:14.265038 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.265001 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.265211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.265144 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.265361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.265332 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngcnf\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-kube-api-access-ngcnf\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.265440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.265397 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.366591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.366556 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.366794 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.366673 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.366853 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.366828 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngcnf\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-kube-api-access-ngcnf\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.366896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.366863 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.367422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.367399 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.367554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.367428 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.369679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.369662 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.386926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.386905 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngcnf\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-kube-api-access-ngcnf\") pod \"payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.494843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.494748 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" event={"ID":"66514a82-468a-4b6f-a5fe-08ac2470d27e","Type":"ContainerStarted","Data":"9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327"} Oct 30 03:51:14.494843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.494827 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:14.495029 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.494849 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" event={"ID":"66514a82-468a-4b6f-a5fe-08ac2470d27e","Type":"ContainerStarted","Data":"b9bf4fe86dddfe64f99829a6031a196cf9cb28f9427d3928666ad48720c6f529"} Oct 30 03:51:14.520271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.520233 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:14.540869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.540821 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" podStartSLOduration=2.540805713 podStartE2EDuration="2.540805713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:14.536220361 +0000 UTC m=+2898.538573561" watchObservedRunningTime="2026-10-30 03:51:14.540805713 +0000 UTC m=+2898.543158914" Oct 30 03:51:14.828819 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:51:14.828790 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd05b266_6b11_4808_a8b1_c20faa4d57b4.slice/crio-f5864673f48f008450ff66f8ffc7e99a4f10abcf0d220da46e43e3f78d45938c WatchSource:0}: Error finding container f5864673f48f008450ff66f8ffc7e99a4f10abcf0d220da46e43e3f78d45938c: Status 404 returned error can't find the container with id f5864673f48f008450ff66f8ffc7e99a4f10abcf0d220da46e43e3f78d45938c Oct 30 03:51:14.828971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.828882 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh"] Oct 30 03:51:14.988526 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:14.988493 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:15.513809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:15.513769 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" event={"ID":"bd05b266-6b11-4808-a8b1-c20faa4d57b4","Type":"ContainerStarted","Data":"8e5750cadc0210be1116d3c87f94d41a1bee971d0a0ce2caa4f45adf408210a8"} Oct 30 03:51:15.514275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:15.513869 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" event={"ID":"bd05b266-6b11-4808-a8b1-c20faa4d57b4","Type":"ContainerStarted","Data":"f5864673f48f008450ff66f8ffc7e99a4f10abcf0d220da46e43e3f78d45938c"} Oct 30 03:51:15.559352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:15.559308 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" podStartSLOduration=1.559296522 podStartE2EDuration="1.559296522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:15.555390853 +0000 UTC m=+2899.557744049" watchObservedRunningTime="2026-10-30 03:51:15.559296522 +0000 UTC m=+2899.561649718" Oct 30 03:51:16.521519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:16.521488 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:16.583226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:16.583197 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:18.234688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:18.234650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:19.769902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.769857 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:19.853091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.853060 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2"] Oct 30 03:51:19.858456 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.858430 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:19.879355 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.879327 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:19.884818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.884791 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e84013\"/\"default-dockercfg-vzsj9\"" Oct 30 03:51:19.885121 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.885097 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e84013\"/\"test-model-e84013-kserve-self-signed-certs\"" Oct 30 03:51:19.885326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.885190 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e84013\"/\"kube-root-ca.crt\"" Oct 30 03:51:19.885326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.885279 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e84013\"/\"openshift-service-ca.crt\"" Oct 30 03:51:19.889727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.889705 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2"] Oct 30 03:51:19.948173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.948141 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrtm\" (UniqueName: \"kubernetes.io/projected/9555fbc5-cff4-4090-947e-b11d9bc1b212-kube-api-access-ndrtm\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:19.948335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.948274 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-home\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:19.948335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.948310 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-model-cache\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:19.948411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.948371 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-tmp-dir\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:19.948490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.948476 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9555fbc5-cff4-4090-947e-b11d9bc1b212-tls-certs\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:19.948531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:19.948497 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-dshm\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.031474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.031398 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:20.050473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.050442 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrtm\" (UniqueName: \"kubernetes.io/projected/9555fbc5-cff4-4090-947e-b11d9bc1b212-kube-api-access-ndrtm\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.050623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.050544 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-home\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.050623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.050581 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-model-cache\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.050778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.050653 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-tmp-dir\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.050778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.050741 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9555fbc5-cff4-4090-947e-b11d9bc1b212-tls-certs\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.050899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.050783 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-dshm\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.050938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.050924 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-home\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.051036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.051015 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-model-cache\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.051075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.051037 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-tmp-dir\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.053315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.053290 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-dshm\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.053439 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.053424 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9555fbc5-cff4-4090-947e-b11d9bc1b212-tls-certs\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.069074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.069050 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrtm\" (UniqueName: \"kubernetes.io/projected/9555fbc5-cff4-4090-947e-b11d9bc1b212-kube-api-access-ndrtm\") pod \"test-model-e84013-kserve-8575bdfd9c-vzbj2\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.135783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.133504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:20.178779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.178527 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:20.451618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.451581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:20.553009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.552923 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:20.556046 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:51:20.556018 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9555fbc5_cff4_4090_947e_b11d9bc1b212.slice/crio-094ee2f4e0ef22ce4514c5ffe3defdb44f70a0b3f4e2a9b861a07b7ace6d110e WatchSource:0}: Error finding container 094ee2f4e0ef22ce4514c5ffe3defdb44f70a0b3f4e2a9b861a07b7ace6d110e: Status 404 returned error can't find the container with id 094ee2f4e0ef22ce4514c5ffe3defdb44f70a0b3f4e2a9b861a07b7ace6d110e Oct 30 03:51:20.556836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:20.556813 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2"] Oct 30 03:51:21.376932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:21.376898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:21.564550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:21.564520 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" event={"ID":"9555fbc5-cff4-4090-947e-b11d9bc1b212","Type":"ContainerStarted","Data":"0c8572fdb925b4e382a529d5683462dc881af19675ba139256e3af46052a5bf5"} Oct 30 03:51:21.564741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:21.564559 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" event={"ID":"9555fbc5-cff4-4090-947e-b11d9bc1b212","Type":"ContainerStarted","Data":"094ee2f4e0ef22ce4514c5ffe3defdb44f70a0b3f4e2a9b861a07b7ace6d110e"} Oct 30 03:51:21.616517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:21.616455 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" podStartSLOduration=2.616441444 podStartE2EDuration="2.616441444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:21.615577449 +0000 UTC m=+2905.617930646" watchObservedRunningTime="2026-10-30 03:51:21.616441444 +0000 UTC m=+2905.618794641" Oct 30 03:51:21.632215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:21.632148 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:51:22.180423 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:22.179156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:22.961777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:22.955245 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:24.560724 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:24.560692 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:24.862741 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:51:24.862643 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:51:25.517064 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:25.517034 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:25.582324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:25.582289 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4"] Oct 30 03:51:25.994862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:25.994775 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:26.866298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:26.866265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:27.494868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:27.494835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:27.530396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:27.530359 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:29.096120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:29.096087 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:30.179497 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.179464 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:30.192451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.192428 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:30.550344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.550311 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:30.583261 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.583227 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" podUID="ccbcf37d-25a6-4a04-9863-974293babad6" containerName="payload-processing" containerID="cri-o://494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051" gracePeriod=25 Oct 30 03:51:30.721997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.721954 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:51:30.912909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.912880 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:51:30.984248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.984205 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-provider-credentials\") pod \"ccbcf37d-25a6-4a04-9863-974293babad6\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " Oct 30 03:51:30.984474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.984264 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-routing-overlay-volume\") pod \"ccbcf37d-25a6-4a04-9863-974293babad6\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " Oct 30 03:51:30.984564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.984513 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-plugins-config-volume\") pod \"ccbcf37d-25a6-4a04-9863-974293babad6\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " Oct 30 03:51:30.985787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.984678 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-kube-api-access-f5cbp\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-kube-api-access-f5cbp\") pod \"ccbcf37d-25a6-4a04-9863-974293babad6\" (UID: \"ccbcf37d-25a6-4a04-9863-974293babad6\") " Oct 30 03:51:30.985787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.985235 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-plugins-config-volume" pod "ccbcf37d-25a6-4a04-9863-974293babad6" (UID: "ccbcf37d-25a6-4a04-9863-974293babad6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:30.985787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.985376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-routing-overlay-volume" pod "ccbcf37d-25a6-4a04-9863-974293babad6" (UID: "ccbcf37d-25a6-4a04-9863-974293babad6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:30.986297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.986247 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:30.986297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.986276 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ccbcf37d-25a6-4a04-9863-974293babad6-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:30.988055 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.988017 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-kube-api-access-f5cbp" pod "ccbcf37d-25a6-4a04-9863-974293babad6" (UID: "ccbcf37d-25a6-4a04-9863-974293babad6"). InnerVolumeSpecName "kube-api-access-f5cbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:30.990823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:30.989501 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-provider-credentials" pod "ccbcf37d-25a6-4a04-9863-974293babad6" (UID: "ccbcf37d-25a6-4a04-9863-974293babad6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:31.087941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.087908 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:31.087941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.087939 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5cbp\" (UniqueName: \"kubernetes.io/projected/ccbcf37d-25a6-4a04-9863-974293babad6-kube-api-access-f5cbp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:31.715548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.715509 4442 generic.go:356] "Generic (PLEG): container finished" podID="ccbcf37d-25a6-4a04-9863-974293babad6" containerID="494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051" exitCode=0 Oct 30 03:51:31.716053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.715558 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" Oct 30 03:51:31.716053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.715585 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" event={"ID":"ccbcf37d-25a6-4a04-9863-974293babad6","Type":"ContainerDied","Data":"494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051"} Oct 30 03:51:31.716053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.715640 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4" event={"ID":"ccbcf37d-25a6-4a04-9863-974293babad6","Type":"ContainerDied","Data":"caf478e56abdbd75bc7027e3feabca5026413e44042bdb494fbd4580fa2b5340"} Oct 30 03:51:31.716053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.715661 4442 scope.go:122] "RemoveContainer" containerID="494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051" Oct 30 03:51:31.727932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.727914 4442 scope.go:122] "RemoveContainer" containerID="494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051" Oct 30 03:51:31.728215 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:51:31.728197 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051\": container with ID starting with 494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051 not found: ID does not exist" containerID="494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051" Oct 30 03:51:31.728274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.728223 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051"} err="failed to get container status \"494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051\": rpc error: code = NotFound desc = could not find container \"494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051\": container with ID starting with 494f96e5b6600827e475450c938f78dce12d94eb189d3b84fedff64dc0f7e051 not found: ID does not exist" Oct 30 03:51:31.764434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.764410 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4"] Oct 30 03:51:31.775289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:31.775268 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-9d96957k4"] Oct 30 03:51:32.108984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:32.108956 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbcf37d-25a6-4a04-9863-974293babad6" path="/var/lib/kubelet/pods/ccbcf37d-25a6-4a04-9863-974293babad6/volumes" Oct 30 03:51:32.120196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:32.120169 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:33.594927 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:33.594891 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:34.992684 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:34.992632 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:35.569419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:35.569386 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh"] Oct 30 03:51:35.626882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:35.626853 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:35.727000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:35.726962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:36.299668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.299637 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:36.466169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.466136 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q"] Oct 30 03:51:36.467381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.467360 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ccbcf37d-25a6-4a04-9863-974293babad6" containerName="payload-processing" Oct 30 03:51:36.467446 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.467385 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbcf37d-25a6-4a04-9863-974293babad6" containerName="payload-processing" Oct 30 03:51:36.467551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.467540 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbcf37d-25a6-4a04-9863-974293babad6" containerName="payload-processing" Oct 30 03:51:36.479302 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.479267 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.533238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.505014 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q"] Oct 30 03:51:36.533238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.508180 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-35fc87\"/\"test-model-35fc87-kserve-self-signed-certs\"" Oct 30 03:51:36.533238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.508291 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-35fc87\"/\"openshift-service-ca.crt\"" Oct 30 03:51:36.533238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.508381 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-35fc87\"/\"kube-root-ca.crt\"" Oct 30 03:51:36.533238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.508521 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-35fc87\"/\"default-dockercfg-6d2j7\"" Oct 30 03:51:36.533554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.533255 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:36.584805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.584718 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tls-certs\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.584805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.584780 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-dshm\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.584989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.584839 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tmp-dir\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.584989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.584955 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zfk2\" (UniqueName: \"kubernetes.io/projected/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-kube-api-access-2zfk2\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.585129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.585114 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-home\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.585262 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.585250 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-model-cache\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.628868 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.628831 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:36.686989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.686955 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-home\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687061 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-model-cache\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687125 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tls-certs\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687157 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687146 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-dshm\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687275 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687163 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tmp-dir\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687324 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zfk2\" (UniqueName: \"kubernetes.io/projected/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-kube-api-access-2zfk2\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687396 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-home\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687416 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-model-cache\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.687665 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.687561 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tmp-dir\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.689676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.689656 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-dshm\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.689818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.689747 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tls-certs\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.739778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.731688 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zfk2\" (UniqueName: \"kubernetes.io/projected/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-kube-api-access-2zfk2\") pod \"test-model-35fc87-kserve-b78678587-dqk2q\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:36.822732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:36.822691 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:37.172358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:37.172326 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q"] Oct 30 03:51:37.172601 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:51:37.172583 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod174ed8a3_1a7f_45b0_816f_b5fd461b6a24.slice/crio-68409f8e24eb77943a15ea37fd47fd192b6c2b45805f26b555badf3611639e31 WatchSource:0}: Error finding container 68409f8e24eb77943a15ea37fd47fd192b6c2b45805f26b555badf3611639e31: Status 404 returned error can't find the container with id 68409f8e24eb77943a15ea37fd47fd192b6c2b45805f26b555badf3611639e31 Oct 30 03:51:37.320807 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:37.320767 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4"] Oct 30 03:51:37.452996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:37.452910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:37.814776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:37.814607 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" event={"ID":"174ed8a3-1a7f-45b0-816f-b5fd461b6a24","Type":"ContainerStarted","Data":"9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41"} Oct 30 03:51:37.814776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:37.814672 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" event={"ID":"174ed8a3-1a7f-45b0-816f-b5fd461b6a24","Type":"ContainerStarted","Data":"68409f8e24eb77943a15ea37fd47fd192b6c2b45805f26b555badf3611639e31"} Oct 30 03:51:37.835616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:37.828192 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:37.870291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:37.870225 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" podStartSLOduration=1.870204099 podStartE2EDuration="1.870204099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:37.86211356 +0000 UTC m=+2921.864466759" watchObservedRunningTime="2026-10-30 03:51:37.870204099 +0000 UTC m=+2921.872557297" Oct 30 03:51:38.217582 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:38.217504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:39.725277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:39.725162 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:40.569967 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.569927 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" podUID="bd05b266-6b11-4808-a8b1-c20faa4d57b4" containerName="payload-processing" containerID="cri-o://8e5750cadc0210be1116d3c87f94d41a1bee971d0a0ce2caa4f45adf408210a8" gracePeriod=25 Oct 30 03:51:40.849342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.849311 4442 generic.go:356] "Generic (PLEG): container finished" podID="bd05b266-6b11-4808-a8b1-c20faa4d57b4" containerID="8e5750cadc0210be1116d3c87f94d41a1bee971d0a0ce2caa4f45adf408210a8" exitCode=0 Oct 30 03:51:40.849695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.849355 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" event={"ID":"bd05b266-6b11-4808-a8b1-c20faa4d57b4","Type":"ContainerDied","Data":"8e5750cadc0210be1116d3c87f94d41a1bee971d0a0ce2caa4f45adf408210a8"} Oct 30 03:51:40.849695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.849416 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" event={"ID":"bd05b266-6b11-4808-a8b1-c20faa4d57b4","Type":"ContainerDied","Data":"f5864673f48f008450ff66f8ffc7e99a4f10abcf0d220da46e43e3f78d45938c"} Oct 30 03:51:40.849695 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.849433 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5864673f48f008450ff66f8ffc7e99a4f10abcf0d220da46e43e3f78d45938c" Oct 30 03:51:40.863093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.863069 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:40.959166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.959134 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-plugins-config-volume\") pod \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " Oct 30 03:51:40.959357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.959229 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-kube-api-access-ngcnf\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-kube-api-access-ngcnf\") pod \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " Oct 30 03:51:40.959357 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.959298 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-routing-overlay-volume\") pod \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " Oct 30 03:51:40.959485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.959380 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-provider-credentials\") pod \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\" (UID: \"bd05b266-6b11-4808-a8b1-c20faa4d57b4\") " Oct 30 03:51:40.959639 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.959614 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-plugins-config-volume" pod "bd05b266-6b11-4808-a8b1-c20faa4d57b4" (UID: "bd05b266-6b11-4808-a8b1-c20faa4d57b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:40.959721 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.959661 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-routing-overlay-volume" pod "bd05b266-6b11-4808-a8b1-c20faa4d57b4" (UID: "bd05b266-6b11-4808-a8b1-c20faa4d57b4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:40.959989 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.959966 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:40.960143 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.960125 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd05b266-6b11-4808-a8b1-c20faa4d57b4-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:40.961607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.961583 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-provider-credentials" pod "bd05b266-6b11-4808-a8b1-c20faa4d57b4" (UID: "bd05b266-6b11-4808-a8b1-c20faa4d57b4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:40.961607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:40.961598 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-kube-api-access-ngcnf" pod "bd05b266-6b11-4808-a8b1-c20faa4d57b4" (UID: "bd05b266-6b11-4808-a8b1-c20faa4d57b4"). InnerVolumeSpecName "kube-api-access-ngcnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:41.061298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:41.061264 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngcnf\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-kube-api-access-ngcnf\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:41.061298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:41.061291 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd05b266-6b11-4808-a8b1-c20faa4d57b4-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:41.308279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:41.308243 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:41.855966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:41.855931 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh" Oct 30 03:51:41.904426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:41.904380 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh"] Oct 30 03:51:41.914735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:41.914709 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-689bcf/payload-processing-external-model-e2e-extmod-a-689bcf-5c99w6fmh"] Oct 30 03:51:42.109884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.109801 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd05b266-6b11-4808-a8b1-c20faa4d57b4" path="/var/lib/kubelet/pods/bd05b266-6b11-4808-a8b1-c20faa4d57b4/volumes" Oct 30 03:51:42.276505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.276477 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:42.321864 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.321804 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" podUID="66514a82-468a-4b6f-a5fe-08ac2470d27e" containerName="payload-processing" containerID="cri-o://9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327" gracePeriod=25 Oct 30 03:51:42.379310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.379217 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:42.603203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.603179 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:42.689075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.688990 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-routing-overlay-volume\") pod \"66514a82-468a-4b6f-a5fe-08ac2470d27e\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " Oct 30 03:51:42.689253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.689099 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-plugins-config-volume\") pod \"66514a82-468a-4b6f-a5fe-08ac2470d27e\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " Oct 30 03:51:42.689253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.689133 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-provider-credentials\") pod \"66514a82-468a-4b6f-a5fe-08ac2470d27e\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " Oct 30 03:51:42.689394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.689342 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-kube-api-access-mdxhr\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-kube-api-access-mdxhr\") pod \"66514a82-468a-4b6f-a5fe-08ac2470d27e\" (UID: \"66514a82-468a-4b6f-a5fe-08ac2470d27e\") " Oct 30 03:51:42.689394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.689368 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-routing-overlay-volume" pod "66514a82-468a-4b6f-a5fe-08ac2470d27e" (UID: "66514a82-468a-4b6f-a5fe-08ac2470d27e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:42.689734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.689712 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-plugins-config-volume" pod "66514a82-468a-4b6f-a5fe-08ac2470d27e" (UID: "66514a82-468a-4b6f-a5fe-08ac2470d27e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:51:42.690103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.690080 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:42.690210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.690109 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66514a82-468a-4b6f-a5fe-08ac2470d27e-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:42.694333 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.694308 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-provider-credentials" pod "66514a82-468a-4b6f-a5fe-08ac2470d27e" (UID: "66514a82-468a-4b6f-a5fe-08ac2470d27e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:42.694475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.694427 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-kube-api-access-mdxhr" pod "66514a82-468a-4b6f-a5fe-08ac2470d27e" (UID: "66514a82-468a-4b6f-a5fe-08ac2470d27e"). InnerVolumeSpecName "kube-api-access-mdxhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:51:42.792590 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.792557 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdxhr\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-kube-api-access-mdxhr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:42.792728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.792594 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66514a82-468a-4b6f-a5fe-08ac2470d27e-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:51:42.800929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.800906 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:42.864129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.864101 4442 generic.go:356] "Generic (PLEG): container finished" podID="66514a82-468a-4b6f-a5fe-08ac2470d27e" containerID="9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327" exitCode=0 Oct 30 03:51:42.864514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.864159 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" Oct 30 03:51:42.864514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.864198 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" event={"ID":"66514a82-468a-4b6f-a5fe-08ac2470d27e","Type":"ContainerDied","Data":"9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327"} Oct 30 03:51:42.864514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.864258 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4" event={"ID":"66514a82-468a-4b6f-a5fe-08ac2470d27e","Type":"ContainerDied","Data":"b9bf4fe86dddfe64f99829a6031a196cf9cb28f9427d3928666ad48720c6f529"} Oct 30 03:51:42.864514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.864287 4442 scope.go:122] "RemoveContainer" containerID="9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327" Oct 30 03:51:42.879057 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.879035 4442 scope.go:122] "RemoveContainer" containerID="9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327" Oct 30 03:51:42.879352 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:51:42.879332 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327\": container with ID starting with 9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327 not found: ID does not exist" containerID="9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327" Oct 30 03:51:42.879426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.879361 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327"} err="failed to get container status \"9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327\": rpc error: code = NotFound desc = could not find container \"9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327\": container with ID starting with 9628a3d64311caca248aaade605e4bc3eb10b94bb95d156311843de0b03b4327 not found: ID does not exist" Oct 30 03:51:42.901486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.901465 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:42.913322 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.913303 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4"] Oct 30 03:51:42.917103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:42.917084 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-35f43f/payload-processing-external-model-e2e-extmod-b-35f43f-69d47snp4"] Oct 30 03:51:43.431306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:43.431264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:43.548712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:43.548676 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:44.110492 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:44.110464 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66514a82-468a-4b6f-a5fe-08ac2470d27e" path="/var/lib/kubelet/pods/66514a82-468a-4b6f-a5fe-08ac2470d27e/volumes" Oct 30 03:51:44.718062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:44.718004 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:46.213959 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.213924 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z"] Oct 30 03:51:46.214662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.214645 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bd05b266-6b11-4808-a8b1-c20faa4d57b4" containerName="payload-processing" Oct 30 03:51:46.214662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.214661 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd05b266-6b11-4808-a8b1-c20faa4d57b4" containerName="payload-processing" Oct 30 03:51:46.214912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.214673 4442 cpu_manager.go:414] "containerMap: removing container" podUID="66514a82-468a-4b6f-a5fe-08ac2470d27e" containerName="payload-processing" Oct 30 03:51:46.214912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.214678 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66514a82-468a-4b6f-a5fe-08ac2470d27e" containerName="payload-processing" Oct 30 03:51:46.214912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.214841 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd05b266-6b11-4808-a8b1-c20faa4d57b4" containerName="payload-processing" Oct 30 03:51:46.214912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.214851 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="66514a82-468a-4b6f-a5fe-08ac2470d27e" containerName="payload-processing" Oct 30 03:51:46.219972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.219950 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.242771 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.242733 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z"] Oct 30 03:51:46.269118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269088 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269231 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269298 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d390a8d-b124-4c7d-8097-aaf88b393175-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269439 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269568 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269697 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269802 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269727 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269872 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269804 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhxwf\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-kube-api-access-zhxwf\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.269940 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.269866 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371245 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371339 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371405 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371447 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d390a8d-b124-4c7d-8097-aaf88b393175-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371507 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371620 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371667 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371637 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-certs\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371852 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371719 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-data\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371890 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.371919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371912 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.372030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.371951 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhxwf\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-kube-api-access-zhxwf\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.372086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.372050 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.372240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.372185 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-credential-socket\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.372403 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.372358 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d390a8d-b124-4c7d-8097-aaf88b393175-istiod-ca-cert\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.374111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.374086 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-envoy\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.374573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.374549 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-podinfo\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.389932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.389911 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhxwf\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-kube-api-access-zhxwf\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.390031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.389953 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-token\") pod \"e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.418657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.418618 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:46.536252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.536112 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:46.823540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.823464 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:46.836942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.836919 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:46.866363 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:51:46.866326 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d390a8d_b124_4c7d_8097_aaf88b393175.slice/crio-ecf3514e25dd9fd570046a0b7bcf7a3c10b7006266a12ed36366e4a964fa721b WatchSource:0}: Error finding container ecf3514e25dd9fd570046a0b7bcf7a3c10b7006266a12ed36366e4a964fa721b: Status 404 returned error can't find the container with id ecf3514e25dd9fd570046a0b7bcf7a3c10b7006266a12ed36366e4a964fa721b Oct 30 03:51:46.866654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.866633 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z"] Oct 30 03:51:46.869315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.869287 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:46.869398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.869349 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:46.869398 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.869377 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:46.897244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.897217 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" event={"ID":"4d390a8d-b124-4c7d-8097-aaf88b393175","Type":"ContainerStarted","Data":"ecf3514e25dd9fd570046a0b7bcf7a3c10b7006266a12ed36366e4a964fa721b"} Oct 30 03:51:46.909457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:46.909433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:51:47.014773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.009665 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm"] Oct 30 03:51:47.018832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.015369 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.034130 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.034090 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm"] Oct 30 03:51:47.085876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.085745 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.085886 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.085946 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.086137 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8kq\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-kube-api-access-sv8kq\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086277 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.086262 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.086287 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.086318 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086428 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.086380 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.086489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.086473 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.188643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.188607 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.188884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.188680 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.188884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.188727 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.188884 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.188808 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.188926 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.188977 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189083 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.189020 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.189091 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.189140 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-data\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.189217 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8kq\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-kube-api-access-sv8kq\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.189358 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.189664 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.189641 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-credential-socket\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.192264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.190278 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istiod-ca-cert\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.192264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.190545 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-certs\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.192470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.192449 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-envoy\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.192470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.192463 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-podinfo\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.214335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.214307 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-token\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.214710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.214453 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8kq\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-kube-api-access-sv8kq\") pod \"e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.348413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.348330 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:47.671381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.671310 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:47.718159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.692062 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm"] Oct 30 03:51:47.718159 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:51:47.698388 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9331b8fa_1b36_48c8_bb92_49ea25d8e66b.slice/crio-69955593565b6bafb4deae1b225e7fcd3837b08d1bd5014c326dd12f564d2a95 WatchSource:0}: Error finding container 69955593565b6bafb4deae1b225e7fcd3837b08d1bd5014c326dd12f564d2a95: Status 404 returned error can't find the container with id 69955593565b6bafb4deae1b225e7fcd3837b08d1bd5014c326dd12f564d2a95 Oct 30 03:51:47.718159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.708159 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:47.718159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.708255 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:47.718159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.708328 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:51:47.780389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.780362 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:47.921880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.921730 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" event={"ID":"9331b8fa-1b36-48c8-bb92-49ea25d8e66b","Type":"ContainerStarted","Data":"f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06"} Oct 30 03:51:47.921880 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.921842 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" event={"ID":"9331b8fa-1b36-48c8-bb92-49ea25d8e66b","Type":"ContainerStarted","Data":"69955593565b6bafb4deae1b225e7fcd3837b08d1bd5014c326dd12f564d2a95"} Oct 30 03:51:47.923835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:47.923805 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" event={"ID":"4d390a8d-b124-4c7d-8097-aaf88b393175","Type":"ContainerStarted","Data":"f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f"} Oct 30 03:51:48.021833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.020026 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podStartSLOduration=2.020003147 podStartE2EDuration="2.020003147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:47.967798616 +0000 UTC m=+2931.970151813" watchObservedRunningTime="2026-10-30 03:51:48.020003147 +0000 UTC m=+2932.022356345" Oct 30 03:51:48.349144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.349114 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:48.350632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.350605 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 03:51:48.350828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.350655 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 03:51:48.442270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.442236 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:48.539837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.536995 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:48.539837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.538121 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 03:51:48.539837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:48.538175 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 03:51:49.348993 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:49.348952 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 03:51:49.349182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:49.349012 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 03:51:49.444922 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:49.444833 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:49.540105 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:49.540072 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 03:51:49.540971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:49.540941 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 03:51:49.541867 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:49.541831 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:50.349805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:50.349579 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 03:51:50.349805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:50.349628 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 03:51:50.411781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:50.408650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:50.511777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:50.511632 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:50.571893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:50.557118 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 03:51:50.571893 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:50.557180 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 03:51:51.321244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:51.321208 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:51.348990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:51.348945 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 03:51:51.349164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:51.349007 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 03:51:51.542963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:51.542927 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 03:51:51.543586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:51.543549 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 03:51:52.341395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:52.341331 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:52.349551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:52.349522 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 03:51:52.349686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:52.349576 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 03:51:52.539779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:52.538099 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" start-of-body= Oct 30 03:51:52.539779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:52.538161 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.203:15021/healthz/ready\": dial tcp 10.134.0.203:15021: connect: connection refused" Oct 30 03:51:53.297780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:53.294455 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:53.349014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:53.348974 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 03:51:53.349198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:53.349028 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 03:51:53.542321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:53.542286 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:53.542805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:53.542780 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:53.543504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:53.543483 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:51:53.608857 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:53.608699 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" podStartSLOduration=7.608682441 podStartE2EDuration="7.608682441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:51:48.022942884 +0000 UTC m=+2932.025296084" watchObservedRunningTime="2026-10-30 03:51:53.608682441 +0000 UTC m=+2937.611035642" Oct 30 03:51:54.004644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:54.004612 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:54.349110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:54.349019 4442 patch_prober.go:29] interesting pod/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" start-of-body= Oct 30 03:51:54.349110 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:54.349077 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.204:15021/healthz/ready\": dial tcp 10.134.0.204:15021: connect: connection refused" Oct 30 03:51:54.862482 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:51:54.862448 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:51:55.214224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:55.214144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:55.329873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:55.329830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:55.355076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:55.355035 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:55.355561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:55.355440 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:55.356222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:55.356201 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:51:55.555875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:55.555846 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:57.005334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:57.005297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:58.584414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:58.584346 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:51:59.981918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:51:59.981889 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:01.406825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:01.406786 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:02.875052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:02.874990 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:04.391388 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:04.391325 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:05.811973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:05.811940 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:07.372297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:07.372265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:08.901897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:08.901859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:09.077999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:09.077960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:09.582155 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:09.582123 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:10.097077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:10.097031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:10.306778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:10.305234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:10.612720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:10.612688 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:12.095089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:12.095056 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:13.771239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:13.771201 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:14.138915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:14.138831 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:14.260708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:14.260677 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:15.506326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:15.506288 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:15.857297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:15.857224 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:15.955325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:15.955297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:17.285577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:17.285548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:18.731783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:18.731718 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:20.194858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:20.194826 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:20.695533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:20.695499 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:52:21.866142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:21.865287 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:23.530795 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:23.530765 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:24.863169 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:52:24.863114 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:52:25.078738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:25.078706 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:26.496925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:26.496888 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:28.062044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:28.062012 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:29.485937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:29.485862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:31.005203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:31.005164 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:32.482479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.482442 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx"] Oct 30 03:52:32.487293 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.487267 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.512723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.512696 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a3d585-serving-cert\"" Oct 30 03:52:32.519196 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.519165 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx"] Oct 30 03:52:32.555199 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.555166 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:32.640618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.640586 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8095a7-85ed-49b2-9676-c7c45e761463-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.640811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.640633 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-metrics-certs\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.640811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.640719 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxt6s\" (UniqueName: \"kubernetes.io/projected/4b8095a7-85ed-49b2-9676-c7c45e761463-kube-api-access-rxt6s\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.640898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.640812 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-maas-api-tls\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.652865 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.652824 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:32.746885 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.746781 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8095a7-85ed-49b2-9676-c7c45e761463-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.747177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.747151 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-metrics-certs\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.747325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.747311 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxt6s\" (UniqueName: \"kubernetes.io/projected/4b8095a7-85ed-49b2-9676-c7c45e761463-kube-api-access-rxt6s\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.747506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.747491 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-maas-api-tls\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.748834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.748746 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8095a7-85ed-49b2-9676-c7c45e761463-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.754830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.753903 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:32.759281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.756991 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-maas-api-tls\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.759281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.759065 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-metrics-certs\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.768894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.768860 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxt6s\" (UniqueName: \"kubernetes.io/projected/4b8095a7-85ed-49b2-9676-c7c45e761463-kube-api-access-rxt6s\") pod \"maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:32.802509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:32.802472 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:33.120326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:33.120292 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx"] Oct 30 03:52:33.120573 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:52:33.120538 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8095a7_85ed_49b2_9676_c7c45e761463.slice/crio-09485e8ff64dd0d905ea0a4baaec5ef149e5aaa2134d98953ab549534b0b4e42 WatchSource:0}: Error finding container 09485e8ff64dd0d905ea0a4baaec5ef149e5aaa2134d98953ab549534b0b4e42: Status 404 returned error can't find the container with id 09485e8ff64dd0d905ea0a4baaec5ef149e5aaa2134d98953ab549534b0b4e42 Oct 30 03:52:33.677445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:33.676218 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" event={"ID":"4b8095a7-85ed-49b2-9676-c7c45e761463","Type":"ContainerStarted","Data":"e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001"} Oct 30 03:52:33.677445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:33.676287 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" event={"ID":"4b8095a7-85ed-49b2-9676-c7c45e761463","Type":"ContainerStarted","Data":"09485e8ff64dd0d905ea0a4baaec5ef149e5aaa2134d98953ab549534b0b4e42"} Oct 30 03:52:33.677445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:33.677375 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:33.723671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:33.723573 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" podStartSLOduration=1.4160658769999999 podStartE2EDuration="1.723560088s" totalImagesPullingTime="307.494211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:33.719579086 +0000 UTC m=+2977.721932302" watchObservedRunningTime="2026-10-30 03:52:33.723560088 +0000 UTC m=+2977.725913284" Oct 30 03:52:34.182405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:34.182363 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:35.694394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:35.694358 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:37.178908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:37.178864 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:38.684418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:38.684383 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:40.086425 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:40.086391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:40.696141 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:40.696114 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 03:52:41.538462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:41.538428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:42.893394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:42.893360 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:44.315693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:44.315660 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:45.909075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:45.909041 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:47.398741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:47.398696 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:48.862154 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:48.862101 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:48.983214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:48.983174 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:49.157238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.157147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:49.186519 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.186487 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm"] Oct 30 03:52:49.192362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.192340 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.206677 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.206632 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm"] Oct 30 03:52:49.353623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.353592 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.353623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.353630 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlwkl\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-kube-api-access-jlwkl\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.353950 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.353924 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.353998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.353966 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.455351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.455270 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlwkl\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-kube-api-access-jlwkl\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.455504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.455484 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.455558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.455511 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.455618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.455601 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.456343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.456315 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.456469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.456352 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.458307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.458288 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.472798 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.472774 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlwkl\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-kube-api-access-jlwkl\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.525192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.525163 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:49.991606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:49.908620 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:50.220780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:50.214854 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm"] Oct 30 03:52:50.607070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:50.606960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:51.042774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:51.042719 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" event={"ID":"53e5aaaa-42e2-4880-b946-c740614396eb","Type":"ContainerStarted","Data":"ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228"} Oct 30 03:52:51.043271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:51.042813 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" event={"ID":"53e5aaaa-42e2-4880-b946-c740614396eb","Type":"ContainerStarted","Data":"5395ddfa81760752f837ea660dfc33d2275a2ecc42d1a7aed5166bcdb29972bd"} Oct 30 03:52:51.043271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:51.042839 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:52:51.083240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:51.083191 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" podStartSLOduration=2.083176301 podStartE2EDuration="2.083176301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:52:51.081293752 +0000 UTC m=+2995.083646950" watchObservedRunningTime="2026-10-30 03:52:51.083176301 +0000 UTC m=+2995.085529498" Oct 30 03:52:51.114601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:51.114518 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:51.532844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:51.532782 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:52.257640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:52.257605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:52.595671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:52.595583 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:53.403953 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:53.403900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:54.040358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:54.040327 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:54.862678 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:52:54.862644 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:52:55.598197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:55.598160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:55.730453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:55.730405 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw"] Oct 30 03:52:55.731875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:55.731843 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" containerID="cri-o://d9abb2e2c4cd416e5f29332be9cac5a7e7c1c178db5577b38e39bf8bee688797" gracePeriod=30 Oct 30 03:52:56.585783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:56.585736 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:52:56.588741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:56.588717 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:52:57.083395 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:57.083358 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:58.493654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:58.493615 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:59.877430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:59.877397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:52:59.991074 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:52:59.990989 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:00.074170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:00.074105 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:01.121165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.121131 4442 generic.go:356] "Generic (PLEG): container finished" podID="966592bb-0715-43e7-aa0e-01f043c68794" containerID="d9abb2e2c4cd416e5f29332be9cac5a7e7c1c178db5577b38e39bf8bee688797" exitCode=0 Oct 30 03:53:01.121524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.121202 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" event={"ID":"966592bb-0715-43e7-aa0e-01f043c68794","Type":"ContainerDied","Data":"d9abb2e2c4cd416e5f29332be9cac5a7e7c1c178db5577b38e39bf8bee688797"} Oct 30 03:53:01.121524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.121254 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" event={"ID":"966592bb-0715-43e7-aa0e-01f043c68794","Type":"ContainerDied","Data":"fc40919a0e925ffdedfca48e6293729e95d5b5f2364b229ca2ec664e991d6f33"} Oct 30 03:53:01.121524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.121273 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc40919a0e925ffdedfca48e6293729e95d5b5f2364b229ca2ec664e991d6f33" Oct 30 03:53:01.139556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.139536 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:53:01.236916 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.236889 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-socket\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.236967 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966592bb-0715-43e7-aa0e-01f043c68794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966592bb-0715-43e7-aa0e-01f043c68794-istiod-ca-cert\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237021 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-certs\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237063 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-envoy\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237125 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-kube-api-access-kn29g\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-kube-api-access-kn29g\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237186 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-socket" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:01.237268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237193 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-istio-token\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-istio-token\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237274 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-credential-socket\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237364 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/966592bb-0715-43e7-aa0e-01f043c68794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966592bb-0715-43e7-aa0e-01f043c68794-istio-podinfo\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237404 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-certs" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:01.237511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237445 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966592bb-0715-43e7-aa0e-01f043c68794-istiod-ca-cert" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:01.237796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237526 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-data\") pod \"966592bb-0715-43e7-aa0e-01f043c68794\" (UID: \"966592bb-0715-43e7-aa0e-01f043c68794\") " Oct 30 03:53:01.237796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.237717 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-credential-socket" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:01.238317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.238230 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-data" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:01.238545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.238476 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.238545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.238498 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/966592bb-0715-43e7-aa0e-01f043c68794-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.238545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.238513 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.238545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.238527 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.238545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.238540 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.240044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.240011 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-envoy" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:01.240368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.240340 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/966592bb-0715-43e7-aa0e-01f043c68794-istio-podinfo" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:53:01.240494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.240411 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-kube-api-access-kn29g" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "kube-api-access-kn29g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:01.241437 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.241412 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-istio-token" pod "966592bb-0715-43e7-aa0e-01f043c68794" (UID: "966592bb-0715-43e7-aa0e-01f043c68794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:01.356288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.340331 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/966592bb-0715-43e7-aa0e-01f043c68794-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.356288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.340374 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn29g\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-kube-api-access-kn29g\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.356288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.340394 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966592bb-0715-43e7-aa0e-01f043c68794-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.356288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.340408 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/966592bb-0715-43e7-aa0e-01f043c68794-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:01.515963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:01.515935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:02.066288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:02.066256 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:53:02.131341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:02.131309 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw" Oct 30 03:53:02.135577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:02.135550 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p"] Oct 30 03:53:02.218622 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:02.218591 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw"] Oct 30 03:53:02.223206 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:02.223181 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-m4svw"] Oct 30 03:53:02.909031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:02.908991 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx"] Oct 30 03:53:03.110564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:03.110530 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:04.111474 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:04.111442 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966592bb-0715-43e7-aa0e-01f043c68794" path="/var/lib/kubelet/pods/966592bb-0715-43e7-aa0e-01f043c68794/volumes" Oct 30 03:53:04.501822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:04.501792 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:05.913568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:05.913540 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:07.136535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.136498 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" podUID="a2ac42f3-5809-466a-bd91-fed7fc181ffc" containerName="payload-processing" containerID="cri-o://806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053" gracePeriod=25 Oct 30 03:53:07.317946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.317917 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:07.484430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.484402 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:53:07.547527 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.547483 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-provider-credentials\") pod \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " Oct 30 03:53:07.547716 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.547666 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-plugins-config-volume\") pod \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " Oct 30 03:53:07.547809 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.547789 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-kube-api-access-l9mms\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-kube-api-access-l9mms\") pod \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " Oct 30 03:53:07.547900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.547884 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-routing-overlay-volume\") pod \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\" (UID: \"a2ac42f3-5809-466a-bd91-fed7fc181ffc\") " Oct 30 03:53:07.548258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.548229 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-plugins-config-volume" pod "a2ac42f3-5809-466a-bd91-fed7fc181ffc" (UID: "a2ac42f3-5809-466a-bd91-fed7fc181ffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:07.548381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.548336 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-routing-overlay-volume" pod "a2ac42f3-5809-466a-bd91-fed7fc181ffc" (UID: "a2ac42f3-5809-466a-bd91-fed7fc181ffc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:07.548638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.548612 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:07.548790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.548644 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ac42f3-5809-466a-bd91-fed7fc181ffc-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:07.550035 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.550013 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-kube-api-access-l9mms" pod "a2ac42f3-5809-466a-bd91-fed7fc181ffc" (UID: "a2ac42f3-5809-466a-bd91-fed7fc181ffc"). InnerVolumeSpecName "kube-api-access-l9mms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:07.550311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.550286 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-provider-credentials" pod "a2ac42f3-5809-466a-bd91-fed7fc181ffc" (UID: "a2ac42f3-5809-466a-bd91-fed7fc181ffc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:07.650891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.650814 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9mms\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-kube-api-access-l9mms\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:07.650891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.650850 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2ac42f3-5809-466a-bd91-fed7fc181ffc-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:07.909800 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:07.909739 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" podUID="c78456d8-5a07-4d22-9e43-0c034a0cd729" containerName="payload-processing" containerID="cri-o://9fd93c3d7826e5b71dfe33ee2c785ab351875a29d6e94ee392656df60319b8e0" gracePeriod=25 Oct 30 03:53:08.180432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.179400 4442 generic.go:356] "Generic (PLEG): container finished" podID="a2ac42f3-5809-466a-bd91-fed7fc181ffc" containerID="806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053" exitCode=0 Oct 30 03:53:08.180432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.179447 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" event={"ID":"a2ac42f3-5809-466a-bd91-fed7fc181ffc","Type":"ContainerDied","Data":"806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053"} Oct 30 03:53:08.180432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.179472 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" Oct 30 03:53:08.180432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.179515 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p" event={"ID":"a2ac42f3-5809-466a-bd91-fed7fc181ffc","Type":"ContainerDied","Data":"b8eada1d297b8427952237d3ed532f70a0445f5f6ecbd1517940bb978fdbffbb"} Oct 30 03:53:08.180432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.179547 4442 scope.go:122] "RemoveContainer" containerID="806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053" Oct 30 03:53:08.182376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.181597 4442 generic.go:356] "Generic (PLEG): container finished" podID="c78456d8-5a07-4d22-9e43-0c034a0cd729" containerID="9fd93c3d7826e5b71dfe33ee2c785ab351875a29d6e94ee392656df60319b8e0" exitCode=0 Oct 30 03:53:08.182376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.181729 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" event={"ID":"c78456d8-5a07-4d22-9e43-0c034a0cd729","Type":"ContainerDied","Data":"9fd93c3d7826e5b71dfe33ee2c785ab351875a29d6e94ee392656df60319b8e0"} Oct 30 03:53:08.218457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.218435 4442 scope.go:122] "RemoveContainer" containerID="806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053" Oct 30 03:53:08.218678 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:08.218661 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053\": container with ID starting with 806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053 not found: ID does not exist" containerID="806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053" Oct 30 03:53:08.218726 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.218696 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053"} err="failed to get container status \"806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053\": rpc error: code = NotFound desc = could not find container \"806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053\": container with ID starting with 806bf4f88b08924ae9f70734248767a737a398f553c53f419869171e1b88c053 not found: ID does not exist" Oct 30 03:53:08.219231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.219212 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:53:08.248818 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.248796 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p"] Oct 30 03:53:08.259344 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.259321 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-677579c4p"] Oct 30 03:53:08.262137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.262119 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-routing-overlay-volume\") pod \"c78456d8-5a07-4d22-9e43-0c034a0cd729\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " Oct 30 03:53:08.262233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.262169 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-provider-credentials\") pod \"c78456d8-5a07-4d22-9e43-0c034a0cd729\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " Oct 30 03:53:08.262233 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.262193 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-kube-api-access-d9n7q\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-kube-api-access-d9n7q\") pod \"c78456d8-5a07-4d22-9e43-0c034a0cd729\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " Oct 30 03:53:08.262352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.262309 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-plugins-config-volume\") pod \"c78456d8-5a07-4d22-9e43-0c034a0cd729\" (UID: \"c78456d8-5a07-4d22-9e43-0c034a0cd729\") " Oct 30 03:53:08.262624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.262510 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-routing-overlay-volume" pod "c78456d8-5a07-4d22-9e43-0c034a0cd729" (UID: "c78456d8-5a07-4d22-9e43-0c034a0cd729"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:08.262737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.262698 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-plugins-config-volume" pod "c78456d8-5a07-4d22-9e43-0c034a0cd729" (UID: "c78456d8-5a07-4d22-9e43-0c034a0cd729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:08.263050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.263028 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:08.263050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.263049 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c78456d8-5a07-4d22-9e43-0c034a0cd729-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:08.264530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.264508 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-kube-api-access-d9n7q" pod "c78456d8-5a07-4d22-9e43-0c034a0cd729" (UID: "c78456d8-5a07-4d22-9e43-0c034a0cd729"). InnerVolumeSpecName "kube-api-access-d9n7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:08.264620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.264537 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-provider-credentials" pod "c78456d8-5a07-4d22-9e43-0c034a0cd729" (UID: "c78456d8-5a07-4d22-9e43-0c034a0cd729"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:08.364128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.364097 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:08.364128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.364127 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9n7q\" (UniqueName: \"kubernetes.io/projected/c78456d8-5a07-4d22-9e43-0c034a0cd729-kube-api-access-d9n7q\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:08.886572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:08.886535 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:09.191075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:09.190948 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" event={"ID":"c78456d8-5a07-4d22-9e43-0c034a0cd729","Type":"ContainerDied","Data":"a0f035cbabb900e526c584cc3c6af793cc4ad383bcbde7ff7e0a852f0c8ad55d"} Oct 30 03:53:09.191075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:09.190973 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx" Oct 30 03:53:09.191075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:09.191016 4442 scope.go:122] "RemoveContainer" containerID="9fd93c3d7826e5b71dfe33ee2c785ab351875a29d6e94ee392656df60319b8e0" Oct 30 03:53:09.239093 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:09.239064 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx"] Oct 30 03:53:09.249453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:09.249423 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-372480/payload-processing-external-model-e2e-extmod-b-372480-565f6w5qx"] Oct 30 03:53:10.109098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:10.109067 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ac42f3-5809-466a-bd91-fed7fc181ffc" path="/var/lib/kubelet/pods/a2ac42f3-5809-466a-bd91-fed7fc181ffc/volumes" Oct 30 03:53:10.109549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:10.109535 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c78456d8-5a07-4d22-9e43-0c034a0cd729" path="/var/lib/kubelet/pods/c78456d8-5a07-4d22-9e43-0c034a0cd729/volumes" Oct 30 03:53:10.375833 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:10.375713 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:11.854032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:11.853996 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:12.486574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.486538 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:12.602119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.602080 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:12.836026 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.835991 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h"] Oct 30 03:53:12.836848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.836823 4442 cpu_manager.go:414] "containerMap: removing container" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" Oct 30 03:53:12.836848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.836844 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" Oct 30 03:53:12.837094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.836890 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a2ac42f3-5809-466a-bd91-fed7fc181ffc" containerName="payload-processing" Oct 30 03:53:12.837094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.836895 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ac42f3-5809-466a-bd91-fed7fc181ffc" containerName="payload-processing" Oct 30 03:53:12.837094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.836907 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c78456d8-5a07-4d22-9e43-0c034a0cd729" containerName="payload-processing" Oct 30 03:53:12.837094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.836911 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c78456d8-5a07-4d22-9e43-0c034a0cd729" containerName="payload-processing" Oct 30 03:53:12.837094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.837055 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="966592bb-0715-43e7-aa0e-01f043c68794" containerName="istio-proxy" Oct 30 03:53:12.837094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.837076 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ac42f3-5809-466a-bd91-fed7fc181ffc" containerName="payload-processing" Oct 30 03:53:12.837094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.837083 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c78456d8-5a07-4d22-9e43-0c034a0cd729" containerName="payload-processing" Oct 30 03:53:12.842579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.842558 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:12.859891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.859853 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h"] Oct 30 03:53:12.947362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.947325 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:12.947362 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.947366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:12.947621 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.947523 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:12.947789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:12.947727 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrfrh\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-kube-api-access-mrfrh\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.049290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.049258 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrfrh\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-kube-api-access-mrfrh\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.049493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.049324 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.049493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.049349 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.049493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.049439 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.050150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.050128 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.050295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.050275 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.051992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.051972 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.067523 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.067493 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrfrh\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-kube-api-access-mrfrh\") pod \"payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.161490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.161178 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:13.719708 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:13.718034 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h"] Oct 30 03:53:14.028927 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.028899 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:14.335401 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.335309 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" event={"ID":"f1732777-7107-463e-ac5e-6cb766beef13","Type":"ContainerStarted","Data":"21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877"} Oct 30 03:53:14.335401 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.335365 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" event={"ID":"f1732777-7107-463e-ac5e-6cb766beef13","Type":"ContainerStarted","Data":"0f96d445c9c7fea1b16b454412b465b42cbf107360c680f0aa7ec7864fff3fcf"} Oct 30 03:53:14.335620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.335414 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:14.372732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.372638 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" podStartSLOduration=2.372618859 podStartE2EDuration="2.372618859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:14.370924841 +0000 UTC m=+3018.373278038" watchObservedRunningTime="2026-10-30 03:53:14.372618859 +0000 UTC m=+3018.374972056" Oct 30 03:53:14.387579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.387539 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:14.415211 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.415162 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h"] Oct 30 03:53:14.418943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.418921 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm"] Oct 30 03:53:14.429572 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:14.429546 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e10c0f/agc-cred-5cfdc2e397bd: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:14.429572 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:14.429573 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:14.429795 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:14.429627 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:14.92960825 +0000 UTC m=+3018.931961431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:14.487624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.487597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:14.572001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:14.571973 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:14.945713 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:14.945667 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e10c0f/agc-cred-5cfdc2e397bd: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:14.946014 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:14.945996 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:14.946187 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:14.946175 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:15.946154886 +0000 UTC m=+3019.948508080 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:15.079650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:15.079619 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:15.342505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:15.342471 4442 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" secret="" err="secret \"payload-processing-external-model-e2e--4f8d4144-dockercfg-wmn86\" not found" Oct 30 03:53:15.454931 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:15.454903 4442 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-plugins-e2e-extmod-a-e10c0f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:15.455224 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:15.454976 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:15.954955687 +0000 UTC m=+3019.957308870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:15.493494 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:15.493453 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:15.690782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:15.690673 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:15.985887 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:15.985726 4442 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-plugins-e2e-extmod-a-e10c0f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:15.985887 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:15.985841 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:16.985820925 +0000 UTC m=+3020.988174121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:15.986306 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:15.986283 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e10c0f/agc-cred-5cfdc2e397bd: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:15.986388 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:15.986309 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:15.986388 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:15.986361 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:17.986343667 +0000 UTC m=+3021.988696857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:17.003522 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:17.003491 4442 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-plugins-e2e-extmod-a-e10c0f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:17.003983 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:17.003573 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:19.003554998 +0000 UTC m=+3023.005908189 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:17.278489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:17.278402 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:18.017501 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:18.017476 4442 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-e10c0f/agc-cred-5cfdc2e397bd: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:18.017501 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:18.017497 4442 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h: secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:18.017979 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:18.017547 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:22.017534302 +0000 UTC m=+3026.019887477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : secret "agc-cred-5cfdc2e397bd" not found Oct 30 03:53:18.863434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:18.863398 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:19.054054 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:19.054023 4442 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-plugins-e2e-extmod-a-e10c0f: configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:19.054440 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:19.054089 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume podName:f1732777-7107-463e-ac5e-6cb766beef13 nodeName:}" failed. No retries permitted until 2026-10-30 03:53:23.054075459 +0000 UTC m=+3027.056428638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" (UID: "f1732777-7107-463e-ac5e-6cb766beef13") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-e10c0f" not found Oct 30 03:53:19.419212 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.419170 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" podUID="53e5aaaa-42e2-4880-b946-c740614396eb" containerName="payload-processing" containerID="cri-o://ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228" gracePeriod=25 Oct 30 03:53:19.696178 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.696150 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:53:19.764938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.764892 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-kube-api-access-jlwkl\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-kube-api-access-jlwkl\") pod \"53e5aaaa-42e2-4880-b946-c740614396eb\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " Oct 30 03:53:19.765124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.764980 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-provider-credentials\") pod \"53e5aaaa-42e2-4880-b946-c740614396eb\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " Oct 30 03:53:19.765124 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.765116 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-plugins-config-volume\") pod \"53e5aaaa-42e2-4880-b946-c740614396eb\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " Oct 30 03:53:19.765303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.765279 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-routing-overlay-volume\") pod \"53e5aaaa-42e2-4880-b946-c740614396eb\" (UID: \"53e5aaaa-42e2-4880-b946-c740614396eb\") " Oct 30 03:53:19.765644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.765557 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-plugins-config-volume" pod "53e5aaaa-42e2-4880-b946-c740614396eb" (UID: "53e5aaaa-42e2-4880-b946-c740614396eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:19.766182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.766146 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-routing-overlay-volume" pod "53e5aaaa-42e2-4880-b946-c740614396eb" (UID: "53e5aaaa-42e2-4880-b946-c740614396eb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:19.766182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.766175 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:19.767327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.767296 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-provider-credentials" pod "53e5aaaa-42e2-4880-b946-c740614396eb" (UID: "53e5aaaa-42e2-4880-b946-c740614396eb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:19.767447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.767385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-kube-api-access-jlwkl" pod "53e5aaaa-42e2-4880-b946-c740614396eb" (UID: "53e5aaaa-42e2-4880-b946-c740614396eb"). InnerVolumeSpecName "kube-api-access-jlwkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:19.809352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.809325 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:53:19.874787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.874406 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlwkl\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-kube-api-access-jlwkl\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:19.874787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.874445 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53e5aaaa-42e2-4880-b946-c740614396eb-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:19.874787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:19.874462 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53e5aaaa-42e2-4880-b946-c740614396eb-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:20.326707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.326676 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:20.440162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.433642 4442 generic.go:356] "Generic (PLEG): container finished" podID="53e5aaaa-42e2-4880-b946-c740614396eb" containerID="ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228" exitCode=0 Oct 30 03:53:20.440162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.433873 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" event={"ID":"53e5aaaa-42e2-4880-b946-c740614396eb","Type":"ContainerDied","Data":"ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228"} Oct 30 03:53:20.440162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.433931 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" event={"ID":"53e5aaaa-42e2-4880-b946-c740614396eb","Type":"ContainerDied","Data":"5395ddfa81760752f837ea660dfc33d2275a2ecc42d1a7aed5166bcdb29972bd"} Oct 30 03:53:20.440162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.433961 4442 scope.go:122] "RemoveContainer" containerID="ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228" Oct 30 03:53:20.440162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.434147 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm" Oct 30 03:53:20.456010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.455982 4442 scope.go:122] "RemoveContainer" containerID="ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228" Oct 30 03:53:20.456394 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:20.456367 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228\": container with ID starting with ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228 not found: ID does not exist" containerID="ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228" Oct 30 03:53:20.456475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.456407 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228"} err="failed to get container status \"ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228\": rpc error: code = NotFound desc = could not find container \"ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228\": container with ID starting with ac3b41bd265152b7ea414786efb98f0c9a8f82a07798e2ec6ef88daba0cf4228 not found: ID does not exist" Oct 30 03:53:20.485478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.485448 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm"] Oct 30 03:53:20.496008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:20.495983 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-6875ftpcm"] Oct 30 03:53:21.350087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.350045 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" podUID="f1732777-7107-463e-ac5e-6cb766beef13" containerName="payload-processing" containerID="cri-o://21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877" gracePeriod=25 Oct 30 03:53:21.456390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.456348 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:21.570805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.559790 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:21.654001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.653976 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:21.734323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.734293 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-kube-api-access-mrfrh\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-kube-api-access-mrfrh\") pod \"f1732777-7107-463e-ac5e-6cb766beef13\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " Oct 30 03:53:21.734551 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.734535 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume\") pod \"f1732777-7107-463e-ac5e-6cb766beef13\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " Oct 30 03:53:21.734633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.734602 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-routing-overlay-volume\") pod \"f1732777-7107-463e-ac5e-6cb766beef13\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " Oct 30 03:53:21.734703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.734682 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials\") pod \"f1732777-7107-463e-ac5e-6cb766beef13\" (UID: \"f1732777-7107-463e-ac5e-6cb766beef13\") " Oct 30 03:53:21.734982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.734952 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume" pod "f1732777-7107-463e-ac5e-6cb766beef13" (UID: "f1732777-7107-463e-ac5e-6cb766beef13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:21.735086 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.735020 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-routing-overlay-volume" pod "f1732777-7107-463e-ac5e-6cb766beef13" (UID: "f1732777-7107-463e-ac5e-6cb766beef13"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:21.735383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.735359 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:21.735383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.735387 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1732777-7107-463e-ac5e-6cb766beef13-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:21.737068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.737042 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-kube-api-access-mrfrh" pod "f1732777-7107-463e-ac5e-6cb766beef13" (UID: "f1732777-7107-463e-ac5e-6cb766beef13"). InnerVolumeSpecName "kube-api-access-mrfrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:21.737143 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.737074 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials" pod "f1732777-7107-463e-ac5e-6cb766beef13" (UID: "f1732777-7107-463e-ac5e-6cb766beef13"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:21.867007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.866901 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:21.867007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.866942 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrfrh\" (UniqueName: \"kubernetes.io/projected/f1732777-7107-463e-ac5e-6cb766beef13-kube-api-access-mrfrh\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:21.978435 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:21.978284 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:22.139563 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.139525 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e5aaaa-42e2-4880-b946-c740614396eb" path="/var/lib/kubelet/pods/53e5aaaa-42e2-4880-b946-c740614396eb/volumes" Oct 30 03:53:22.208637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.208399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:22.440336 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.438510 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749"] Oct 30 03:53:22.443803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.441555 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f1732777-7107-463e-ac5e-6cb766beef13" containerName="payload-processing" Oct 30 03:53:22.443803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.441581 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1732777-7107-463e-ac5e-6cb766beef13" containerName="payload-processing" Oct 30 03:53:22.443803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.441639 4442 cpu_manager.go:414] "containerMap: removing container" podUID="53e5aaaa-42e2-4880-b946-c740614396eb" containerName="payload-processing" Oct 30 03:53:22.443803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.441648 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e5aaaa-42e2-4880-b946-c740614396eb" containerName="payload-processing" Oct 30 03:53:22.443803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.441831 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1732777-7107-463e-ac5e-6cb766beef13" containerName="payload-processing" Oct 30 03:53:22.443803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.441863 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e5aaaa-42e2-4880-b946-c740614396eb" containerName="payload-processing" Oct 30 03:53:22.464775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.455348 4442 generic.go:356] "Generic (PLEG): container finished" podID="f1732777-7107-463e-ac5e-6cb766beef13" containerID="21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877" exitCode=0 Oct 30 03:53:22.464775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.455457 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" Oct 30 03:53:22.464775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.456661 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" event={"ID":"f1732777-7107-463e-ac5e-6cb766beef13","Type":"ContainerDied","Data":"21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877"} Oct 30 03:53:22.464775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.456703 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h" event={"ID":"f1732777-7107-463e-ac5e-6cb766beef13","Type":"ContainerDied","Data":"0f96d445c9c7fea1b16b454412b465b42cbf107360c680f0aa7ec7864fff3fcf"} Oct 30 03:53:22.464775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.456734 4442 scope.go:122] "RemoveContainer" containerID="21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877" Oct 30 03:53:22.464775 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.456891 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.483135 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.483032 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:22.483691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.483671 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0393ed-openshift-default-dockercfg-wtkkm\"" Oct 30 03:53:22.509266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.509051 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749"] Oct 30 03:53:22.524589 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.524388 4442 scope.go:122] "RemoveContainer" containerID="21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877" Oct 30 03:53:22.524881 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:22.524851 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877\": container with ID starting with 21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877 not found: ID does not exist" containerID="21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877" Oct 30 03:53:22.524995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.524892 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877"} err="failed to get container status \"21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877\": rpc error: code = NotFound desc = could not find container \"21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877\": container with ID starting with 21a5ed63e666d8707532a72f8c229cfc656b85f88c8ef87bedeb37f384278877 not found: ID does not exist" Oct 30 03:53:22.587723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.587689 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h"] Oct 30 03:53:22.591739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.591712 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-e10c0f/payload-processing-external-model-e2e-extmod-a-e10c0f-56dblg65h"] Oct 30 03:53:22.593571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.593547 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfdm\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-kube-api-access-wmfdm\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.593712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.593693 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-credential-socket\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.593848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.593831 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-certs\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.593946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.593897 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-token\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.594082 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.594059 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-podinfo\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.594189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.594170 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-data\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.594269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.594252 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-socket\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.594350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.594312 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istiod-ca-cert\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.594390 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.594378 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-envoy\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.605213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.605187 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.724450 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-socket\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.724509 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istiod-ca-cert\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.724552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-envoy\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.724594 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfdm\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-kube-api-access-wmfdm\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.724678 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-credential-socket\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.732541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-socket\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.736465 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-certs\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.736543 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-token\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.736617 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-podinfo\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.736710 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-data\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.737235 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-data\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.739394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.737496 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-credential-socket\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.743778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.741347 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-certs\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.743778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.742041 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istiod-ca-cert\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.755104 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.755071 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-envoy\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.767797 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.756205 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-podinfo\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.791711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.785789 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-token\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.800784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.799060 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfdm\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-kube-api-access-wmfdm\") pod \"e2e-inf-a-0393ed-openshift-default-55b9654d67-94749\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:22.811778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:22.805517 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:23.202089 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.202054 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749"] Oct 30 03:53:23.203562 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:53:23.203522 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e37dd5f_96b4_41ae_8e7b_c6ce51cc1a8b.slice/crio-1671cea5726b5f66b4c3b1bf44563316ab4a4de122a7860b6e2730a90d7aa767 WatchSource:0}: Error finding container 1671cea5726b5f66b4c3b1bf44563316ab4a4de122a7860b6e2730a90d7aa767: Status 404 returned error can't find the container with id 1671cea5726b5f66b4c3b1bf44563316ab4a4de122a7860b6e2730a90d7aa767 Oct 30 03:53:23.206741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.206709 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:23.206848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.206798 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:23.206848 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.206831 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:23.467174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.467072 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" event={"ID":"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b","Type":"ContainerStarted","Data":"8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9"} Oct 30 03:53:23.467174 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.467131 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" event={"ID":"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b","Type":"ContainerStarted","Data":"1671cea5726b5f66b4c3b1bf44563316ab4a4de122a7860b6e2730a90d7aa767"} Oct 30 03:53:23.513018 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.512966 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podStartSLOduration=1.512952586 podStartE2EDuration="1.512952586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:23.507332921 +0000 UTC m=+3027.509686118" watchObservedRunningTime="2026-10-30 03:53:23.512952586 +0000 UTC m=+3027.515305782" Oct 30 03:53:23.811309 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:23.811273 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:24.006068 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:24.006037 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:24.124693 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:24.124592 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1732777-7107-463e-ac5e-6cb766beef13" path="/var/lib/kubelet/pods/f1732777-7107-463e-ac5e-6cb766beef13/volumes" Oct 30 03:53:24.817957 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:24.817927 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:24.862858 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:24.862823 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:53:25.061316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.061283 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:25.282449 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.282413 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6"] Oct 30 03:53:25.287295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.287273 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.305735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.305712 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9e0d09-openshift-default-dockercfg-kfh5d\"" Oct 30 03:53:25.318006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.317984 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6"] Oct 30 03:53:25.394676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.394645 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.394882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.394693 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.394933 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.394909 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.394998 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.394980 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.395045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.395021 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e036dea4-c6a2-4956-a75c-8f5f0904d287-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.395134 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.395112 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdbr\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-kube-api-access-swdbr\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.395204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.395185 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.395269 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.395232 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.395312 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.395296 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.483284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.483251 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:25.484443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.484416 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:53:25.496633 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.496605 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e036dea4-c6a2-4956-a75c-8f5f0904d287-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.496787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.496649 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdbr\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-kube-api-access-swdbr\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.496787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.496691 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.496787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.496717 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.496938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.496787 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.496938 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.496908 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.497015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.496952 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.497142 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.497120 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.497229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.497220 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.497350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.497330 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.497452 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.497429 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e036dea4-c6a2-4956-a75c-8f5f0904d287-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.497530 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.497516 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.497584 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.497565 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.498062 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.498037 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.499927 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.499904 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.500129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.500111 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.520031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.520004 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.520373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.520352 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdbr\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-kube-api-access-swdbr\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.531783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.531742 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7"] Oct 30 03:53:25.531992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.531960 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" containerID="cri-o://e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7" gracePeriod=30 Oct 30 03:53:25.595827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.595118 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:25.603323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.603288 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:25.781151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.781122 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:25.902835 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.897263 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:25.984697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.984670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:25.987915 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:53:25.987873 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode036dea4_c6a2_4956_a75c_8f5f0904d287.slice/crio-7654aaa399e8eb22ea7d23c18ed80c984cb48ddc3f99f9f0b17aaf1856d6ec89 WatchSource:0}: Error finding container 7654aaa399e8eb22ea7d23c18ed80c984cb48ddc3f99f9f0b17aaf1856d6ec89: Status 404 returned error can't find the container with id 7654aaa399e8eb22ea7d23c18ed80c984cb48ddc3f99f9f0b17aaf1856d6ec89 Oct 30 03:53:25.988016 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.987923 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6"] Oct 30 03:53:25.991620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.991589 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:25.991712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.991660 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:25.991712 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:25.991693 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:26.143320 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:26.143232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:26.310676 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:26.310622 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:26.533834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:26.531499 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" event={"ID":"e036dea4-c6a2-4956-a75c-8f5f0904d287","Type":"ContainerStarted","Data":"a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4"} Oct 30 03:53:26.533834 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:26.531559 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" event={"ID":"e036dea4-c6a2-4956-a75c-8f5f0904d287","Type":"ContainerStarted","Data":"7654aaa399e8eb22ea7d23c18ed80c984cb48ddc3f99f9f0b17aaf1856d6ec89"} Oct 30 03:53:26.575982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:26.575864 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podStartSLOduration=1.575843247 podStartE2EDuration="1.575843247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:26.572165152 +0000 UTC m=+3030.574518340" watchObservedRunningTime="2026-10-30 03:53:26.575843247 +0000 UTC m=+3030.578196446" Oct 30 03:53:26.604450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:26.604409 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:27.608779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.608040 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 03:53:27.608779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.608122 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 03:53:27.611376 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.611272 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:27.705776 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.705649 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:27.760561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.760530 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf"] Oct 30 03:53:27.765429 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.765409 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:27.781710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.781679 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf"] Oct 30 03:53:27.807778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.794405 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0393ed\"" Oct 30 03:53:27.807778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.794571 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0393ed-dockercfg-8vhd2\"" Oct 30 03:53:27.817738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.817702 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:27.909837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.908493 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9284g\" (UniqueName: \"kubernetes.io/projected/f415ec38-1e43-40b9-868d-034586bad482-kube-api-access-9284g\") pod \"payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:27.909837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:27.908546 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f415ec38-1e43-40b9-868d-034586bad482-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:28.118781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.114682 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9284g\" (UniqueName: \"kubernetes.io/projected/f415ec38-1e43-40b9-868d-034586bad482-kube-api-access-9284g\") pod \"payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:28.118781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.114738 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f415ec38-1e43-40b9-868d-034586bad482-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:28.118781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.115992 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f415ec38-1e43-40b9-868d-034586bad482-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:28.242892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.232355 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9284g\" (UniqueName: \"kubernetes.io/projected/f415ec38-1e43-40b9-868d-034586bad482-kube-api-access-9284g\") pod \"payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:28.402823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.402555 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:28.607814 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.607773 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 03:53:28.608033 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.607847 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 03:53:28.713610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:28.713581 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf"] Oct 30 03:53:28.715195 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:53:28.715167 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf415ec38_1e43_40b9_868d_034586bad482.slice/crio-f39118eedf795fb8136ce26ce87e1a919be9b27eedac4fd4661f4a32214c12e0 WatchSource:0}: Error finding container f39118eedf795fb8136ce26ce87e1a919be9b27eedac4fd4661f4a32214c12e0: Status 404 returned error can't find the container with id f39118eedf795fb8136ce26ce87e1a919be9b27eedac4fd4661f4a32214c12e0 Oct 30 03:53:29.562103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:29.562067 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" event={"ID":"f415ec38-1e43-40b9-868d-034586bad482","Type":"ContainerStarted","Data":"00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208"} Oct 30 03:53:29.562297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:29.562124 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" event={"ID":"f415ec38-1e43-40b9-868d-034586bad482","Type":"ContainerStarted","Data":"f39118eedf795fb8136ce26ce87e1a919be9b27eedac4fd4661f4a32214c12e0"} Oct 30 03:53:29.562297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:29.562165 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:29.563692 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:29.563664 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:29.604778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:29.602127 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" podStartSLOduration=2.602109527 podStartE2EDuration="2.602109527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:29.601989585 +0000 UTC m=+3033.604342784" watchObservedRunningTime="2026-10-30 03:53:29.602109527 +0000 UTC m=+3033.604462727" Oct 30 03:53:29.607960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:29.607929 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 03:53:29.608102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:29.608001 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 03:53:30.607780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:30.606986 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 03:53:30.607780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:30.607062 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 03:53:30.607780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:30.607484 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 03:53:30.607780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:30.607527 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 03:53:30.860876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:30.860847 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:53:31.004779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:30.982076 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-data\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.005097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.005069 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-istio-token\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.005321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.005304 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-socket\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.005485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.005468 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-credential-socket\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.012843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.012806 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-kube-api-access-4tfjn\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-kube-api-access-4tfjn\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.013043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.003113 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-data" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:31.013131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.013001 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-socket" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:31.013266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.013243 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-credential-socket" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:31.013467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.013437 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6a5b40-7745-480e-b79c-69ea11956806-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a5b40-7745-480e-b79c-69ea11956806-istiod-ca-cert\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.013616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.013533 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-certs\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.013616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.013591 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-envoy\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.013729 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.013672 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c6a5b40-7745-480e-b79c-69ea11956806-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a5b40-7745-480e-b79c-69ea11956806-istio-podinfo\") pod \"7c6a5b40-7745-480e-b79c-69ea11956806\" (UID: \"7c6a5b40-7745-480e-b79c-69ea11956806\") " Oct 30 03:53:31.014461 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.014438 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.014548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.014471 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.014548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.014486 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.024774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.016616 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-certs" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:31.024774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.018256 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6a5b40-7745-480e-b79c-69ea11956806-istiod-ca-cert" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:53:31.024774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.023102 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-envoy" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:53:31.035192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.035166 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-kube-api-access-4tfjn" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "kube-api-access-4tfjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:31.035813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.035793 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-istio-token" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:53:31.037223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.037206 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6a5b40-7745-480e-b79c-69ea11956806-istio-podinfo" pod "7c6a5b40-7745-480e-b79c-69ea11956806" (UID: "7c6a5b40-7745-480e-b79c-69ea11956806"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:53:31.116670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.116577 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tfjn\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-kube-api-access-4tfjn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.116670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.116609 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6a5b40-7745-480e-b79c-69ea11956806-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.116670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.116620 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.116670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.116630 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6a5b40-7745-480e-b79c-69ea11956806-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.116670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.116638 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6a5b40-7745-480e-b79c-69ea11956806-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.116670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.116648 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6a5b40-7745-480e-b79c-69ea11956806-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:53:31.211782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.205234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:31.309283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.309247 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:31.407839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.404394 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:31.600778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.600251 4442 generic.go:356] "Generic (PLEG): container finished" podID="7c6a5b40-7745-480e-b79c-69ea11956806" containerID="e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7" exitCode=0 Oct 30 03:53:31.600778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.600499 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" event={"ID":"7c6a5b40-7745-480e-b79c-69ea11956806","Type":"ContainerDied","Data":"e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7"} Oct 30 03:53:31.600778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.600554 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" event={"ID":"7c6a5b40-7745-480e-b79c-69ea11956806","Type":"ContainerDied","Data":"7f5dc2113a3b94abfa9f1541ecee9e6f7487c89857bb14ce5ffb98490d07b201"} Oct 30 03:53:31.600778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.600584 4442 scope.go:122] "RemoveContainer" containerID="e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7" Oct 30 03:53:31.600778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.600775 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7" Oct 30 03:53:31.625783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.617471 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 03:53:31.625783 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.617527 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 03:53:31.736924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.731916 4442 scope.go:122] "RemoveContainer" containerID="e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7" Oct 30 03:53:31.737920 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:31.737303 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7\": container with ID starting with e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7 not found: ID does not exist" containerID="e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7" Oct 30 03:53:31.737920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.737345 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7"} err="failed to get container status \"e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7\": rpc error: code = NotFound desc = could not find container \"e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7\": container with ID starting with e33b6ecba986039dfec43c6f1f392ce8d82acfbb8a2e3f944b467d789f3357a7 not found: ID does not exist" Oct 30 03:53:31.778847 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.778812 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7"] Oct 30 03:53:31.808166 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:31.808121 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-jpdz7"] Oct 30 03:53:32.116662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:32.116618 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" path="/var/lib/kubelet/pods/7c6a5b40-7745-480e-b79c-69ea11956806/volumes" Oct 30 03:53:32.607200 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:32.607156 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 03:53:32.607507 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:32.607477 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 03:53:32.895445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:32.895345 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:32.990567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:32.990534 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:33.069630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.069596 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2"] Oct 30 03:53:33.070396 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.070380 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" Oct 30 03:53:33.070454 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.070398 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" Oct 30 03:53:33.070522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.070514 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6a5b40-7745-480e-b79c-69ea11956806" containerName="istio-proxy" Oct 30 03:53:33.075621 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.075605 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.091267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.091241 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2"] Oct 30 03:53:33.097284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.097138 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4206d9\"" Oct 30 03:53:33.097284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.097267 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4206d9-dockercfg-gz79m\"" Oct 30 03:53:33.203845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.202616 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c811b25f-c31f-464c-8e3f-31c3ef046df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.203845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.202869 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c28s8\" (UniqueName: \"kubernetes.io/projected/c811b25f-c31f-464c-8e3f-31c3ef046df4-kube-api-access-c28s8\") pod \"payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.203845 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.203435 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2"] Oct 30 03:53:33.212918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.212335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:33.244048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.244019 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.309137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.309106 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef298974-3793-440a-ac80-95d7dc13c24a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.309545 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.309476 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgq5v\" (UniqueName: \"kubernetes.io/projected/ef298974-3793-440a-ac80-95d7dc13c24a-kube-api-access-kgq5v\") pod \"payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.309832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.309734 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c811b25f-c31f-464c-8e3f-31c3ef046df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.310191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.310161 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c28s8\" (UniqueName: \"kubernetes.io/projected/c811b25f-c31f-464c-8e3f-31c3ef046df4-kube-api-access-c28s8\") pod \"payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.310789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.310610 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c811b25f-c31f-464c-8e3f-31c3ef046df4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.312736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.312718 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2"] Oct 30 03:53:33.356593 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.348746 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c28s8\" (UniqueName: \"kubernetes.io/projected/c811b25f-c31f-464c-8e3f-31c3ef046df4-kube-api-access-c28s8\") pod \"payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.398787 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.398052 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:33.520670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.515976 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef298974-3793-440a-ac80-95d7dc13c24a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.520670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.516096 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgq5v\" (UniqueName: \"kubernetes.io/projected/ef298974-3793-440a-ac80-95d7dc13c24a-kube-api-access-kgq5v\") pod \"payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.520670 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.517566 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef298974-3793-440a-ac80-95d7dc13c24a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.537204 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.536166 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgq5v\" (UniqueName: \"kubernetes.io/projected/ef298974-3793-440a-ac80-95d7dc13c24a-kube-api-access-kgq5v\") pod \"payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.614487 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.614456 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:33.622932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.622906 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:33.625943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.625919 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:33.626602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.626584 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 03:53:33.830825 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:53:33.830793 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc811b25f_c31f_464c_8e3f_31c3ef046df4.slice/crio-ef7a2f7efb13d1e51c3363f53c2c75c99f6df89f09bf9ba0899d6db2dfb4d881 WatchSource:0}: Error finding container ef7a2f7efb13d1e51c3363f53c2c75c99f6df89f09bf9ba0899d6db2dfb4d881: Status 404 returned error can't find the container with id ef7a2f7efb13d1e51c3363f53c2c75c99f6df89f09bf9ba0899d6db2dfb4d881 Oct 30 03:53:33.832918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.832894 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2"] Oct 30 03:53:33.949935 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:33.949905 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2"] Oct 30 03:53:33.950349 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:53:33.950221 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef298974_3793_440a_ac80_95d7dc13c24a.slice/crio-0f9e1e2a4620975a34be33ec70e70833d49d88c195cad7b61225e3998136d528 WatchSource:0}: Error finding container 0f9e1e2a4620975a34be33ec70e70833d49d88c195cad7b61225e3998136d528: Status 404 returned error can't find the container with id 0f9e1e2a4620975a34be33ec70e70833d49d88c195cad7b61225e3998136d528 Oct 30 03:53:34.641270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.641231 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" event={"ID":"ef298974-3793-440a-ac80-95d7dc13c24a","Type":"ContainerStarted","Data":"6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff"} Oct 30 03:53:34.641475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.641303 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:34.641475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.641324 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" event={"ID":"ef298974-3793-440a-ac80-95d7dc13c24a","Type":"ContainerStarted","Data":"0f9e1e2a4620975a34be33ec70e70833d49d88c195cad7b61225e3998136d528"} Oct 30 03:53:34.643347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.643313 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" event={"ID":"c811b25f-c31f-464c-8e3f-31c3ef046df4","Type":"ContainerStarted","Data":"f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727"} Oct 30 03:53:34.643468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.643367 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" event={"ID":"c811b25f-c31f-464c-8e3f-31c3ef046df4","Type":"ContainerStarted","Data":"ef7a2f7efb13d1e51c3363f53c2c75c99f6df89f09bf9ba0899d6db2dfb4d881"} Oct 30 03:53:34.643548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.643490 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:34.681310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.681262 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" podStartSLOduration=1.681249401 podStartE2EDuration="1.681249401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:34.678246437 +0000 UTC m=+3038.680599635" watchObservedRunningTime="2026-10-30 03:53:34.681249401 +0000 UTC m=+3038.683602597" Oct 30 03:53:34.715370 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.715321 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" podStartSLOduration=1.715307836 podStartE2EDuration="1.715307836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:34.711644681 +0000 UTC m=+3038.713997881" watchObservedRunningTime="2026-10-30 03:53:34.715307836 +0000 UTC m=+3038.717661032" Oct 30 03:53:34.903983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:34.903889 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:36.394247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:36.394214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:37.801050 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:37.801019 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:39.183600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:39.183571 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:40.572860 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:40.572823 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 03:53:40.621637 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:40.621604 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:42.184259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:42.184224 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:43.678163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:43.678130 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:44.835796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.835766 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8"] Oct 30 03:53:44.840656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.840628 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.866290 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.865325 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d53363-openshift-default-dockercfg-bh4js\"" Oct 30 03:53:44.872316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.872278 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8"] Oct 30 03:53:44.930567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930532 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-envoy\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.930567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930570 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-certs\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.930793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930591 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbb56a-d834-41fa-b559-09e8be196c7e-istiod-ca-cert\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.930793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930747 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dh4p\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-kube-api-access-4dh4p\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.930887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930846 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-data\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.930887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930879 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-credential-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.930968 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930947 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-token\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.931013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.930976 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:44.931165 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:44.931145 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-podinfo\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.033669 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.033636 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-podinfo\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.033930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.033780 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-envoy\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.033930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.033814 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-certs\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.033930 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.033910 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbb56a-d834-41fa-b559-09e8be196c7e-istiod-ca-cert\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.034133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034041 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dh4p\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-kube-api-access-4dh4p\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.034133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034102 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-data\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.034253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034132 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-credential-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.034253 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034203 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-token\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.034401 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034376 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-certs\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.034827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034600 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.035076 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034734 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-credential-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.035148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034831 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-data\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.035148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.034953 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-socket\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.035260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.035240 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbb56a-d834-41fa-b559-09e8be196c7e-istiod-ca-cert\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.036459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.036436 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-podinfo\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.036659 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.036640 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-envoy\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.052356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.052324 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-token\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.052465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.052401 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dh4p\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-kube-api-access-4dh4p\") pod \"e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.088992 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.088923 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:45.170263 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.169056 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:45.505097 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:53:45.505069 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfbb56a_d834_41fa_b559_09e8be196c7e.slice/crio-64b869aabfc90d33409a6a94d533983f52d0d911b0e6dd8a2c7769a22bcad707 WatchSource:0}: Error finding container 64b869aabfc90d33409a6a94d533983f52d0d911b0e6dd8a2c7769a22bcad707: Status 404 returned error can't find the container with id 64b869aabfc90d33409a6a94d533983f52d0d911b0e6dd8a2c7769a22bcad707 Oct 30 03:53:45.505687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.505611 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8"] Oct 30 03:53:45.508180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.508151 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:45.508266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.508216 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:45.508266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.508246 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:53:45.653147 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.653116 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 03:53:45.653328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.653200 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 03:53:45.749806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.749178 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" event={"ID":"8cfbb56a-d834-41fa-b559-09e8be196c7e","Type":"ContainerStarted","Data":"5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129"} Oct 30 03:53:45.749806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.749241 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" event={"ID":"8cfbb56a-d834-41fa-b559-09e8be196c7e","Type":"ContainerStarted","Data":"64b869aabfc90d33409a6a94d533983f52d0d911b0e6dd8a2c7769a22bcad707"} Oct 30 03:53:45.790194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:45.790091 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podStartSLOduration=1.790077561 podStartE2EDuration="1.790077561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:45.788173051 +0000 UTC m=+3049.790526282" watchObservedRunningTime="2026-10-30 03:53:45.790077561 +0000 UTC m=+3049.792430759" Oct 30 03:53:46.172369 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:46.172269 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:46.173813 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:46.173785 4442 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 03:53:46.173958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:46.173834 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 03:53:46.722942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:46.722911 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:47.172226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:47.172190 4442 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 03:53:47.172405 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:47.172241 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 03:53:48.173102 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:48.173069 4442 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 03:53:48.173503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:48.173124 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 03:53:48.257219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:48.257171 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:49.172535 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:49.172497 4442 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 03:53:49.172718 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:49.172553 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 03:53:49.808780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:49.808718 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:50.173072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:50.172996 4442 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 03:53:50.173072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:50.173048 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 03:53:50.734092 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:50.734055 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:51.172839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:51.172798 4442 patch_prober.go:29] interesting pod/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 03:53:51.173298 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:51.172862 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 03:53:51.192331 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:51.192297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:51.706528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:51.706485 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:51.819225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:51.819187 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:52.222315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:52.222273 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:52.222970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:52.222640 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:52.227031 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:52.227005 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:53:53.162236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:53.162198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:53.526488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:53.526456 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:54.689155 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:54.689122 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:54.866222 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:53:54.866178 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:53:56.165158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:56.165086 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:56.786569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:56.786526 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:57.266318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:57.266286 4442 scope.go:122] "RemoveContainer" containerID="d9abb2e2c4cd416e5f29332be9cac5a7e7c1c178db5577b38e39bf8bee688797" Oct 30 03:53:57.595518 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:57.595436 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:58.851740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:58.851703 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:53:59.295995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:53:59.295960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:00.976843 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:00.976787 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:02.449571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:02.449537 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:03.907070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:03.907036 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:05.398407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:05.398368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:06.882735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:06.882698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:08.407434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:08.407399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:09.807479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:09.807438 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:10.410801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.410771 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25"] Oct 30 03:54:10.416883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.416857 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.440419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.440387 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2"] Oct 30 03:54:10.445830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.445801 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.453374 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.453347 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25"] Oct 30 03:54:10.464238 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.464209 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2"] Oct 30 03:54:10.494326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.494291 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.494488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.494337 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cw87\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-kube-api-access-8cw87\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.494539 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.494487 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.494586 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.494557 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.494704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.494675 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89464194-c498-4655-8868-a930de543542-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.494869 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.494850 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.494919 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.494901 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.495043 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495010 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.495159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495131 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.495241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495212 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.495387 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495368 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.495493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495475 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49285b-876e-4a6d-8c3e-c8602687306d-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.495567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495515 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.495567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495544 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.495717 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.495693 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.496252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.496226 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.496359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.496283 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89464194-c498-4655-8868-a930de543542-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.496424 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.496383 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7mv\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-kube-api-access-ms7mv\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598226 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598192 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598242 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49285b-876e-4a6d-8c3e-c8602687306d-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598270 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598290 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598310 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598327 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598343 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89464194-c498-4655-8868-a930de543542-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598369 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7mv\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-kube-api-access-ms7mv\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598401 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598427 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cw87\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-kube-api-access-8cw87\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598522 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598591 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598647 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89464194-c498-4655-8868-a930de543542-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598702 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598725 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598743 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598779 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598831 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598893 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.598984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.598955 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599046 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599159 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599214 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89464194-c498-4655-8868-a930de543542-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599279 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49285b-876e-4a6d-8c3e-c8602687306d-istiod-ca-cert\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599489 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-data\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599532 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-certs\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599551 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-credential-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.599666 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.599618 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-socket\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.601548 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.601521 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.601739 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.601719 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.602150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.602129 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89464194-c498-4655-8868-a930de543542-istio-podinfo\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.602636 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.602616 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-envoy\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.616540 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.616508 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cw87\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-kube-api-access-8cw87\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.616991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.616967 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2\" (UID: \"89464194-c498-4655-8868-a930de543542\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:10.617123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.617108 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7mv\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-kube-api-access-ms7mv\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.617270 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.617252 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-token\") pod \"e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.733237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.733163 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:10.774431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:10.774391 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:11.052710 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:54:11.052676 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf49285b_876e_4a6d_8c3e_c8602687306d.slice/crio-f880c69cf5d6c5444bf323bed347c7d069c59e1203549b62e910e2b34c78cc5d WatchSource:0}: Error finding container f880c69cf5d6c5444bf323bed347c7d069c59e1203549b62e910e2b34c78cc5d: Status 404 returned error can't find the container with id f880c69cf5d6c5444bf323bed347c7d069c59e1203549b62e910e2b34c78cc5d Oct 30 03:54:11.053274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.053241 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25"] Oct 30 03:54:11.055778 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.055727 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:54:11.055875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.055814 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:54:11.055875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.055844 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:54:11.091187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.091155 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" event={"ID":"bf49285b-876e-4a6d-8c3e-c8602687306d","Type":"ContainerStarted","Data":"f880c69cf5d6c5444bf323bed347c7d069c59e1203549b62e910e2b34c78cc5d"} Oct 30 03:54:11.110232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.110208 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2"] Oct 30 03:54:11.121714 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:54:11.121684 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89464194_c498_4655_8868_a930de543542.slice/crio-061cb29f246bb3d53e3f48c4b744d100d272c46a8ca71395af633e375bee4b62 WatchSource:0}: Error finding container 061cb29f246bb3d53e3f48c4b744d100d272c46a8ca71395af633e375bee4b62: Status 404 returned error can't find the container with id 061cb29f246bb3d53e3f48c4b744d100d272c46a8ca71395af633e375bee4b62 Oct 30 03:54:11.124647 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.124615 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:54:11.124774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.124682 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:54:11.124774 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.124709 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:54:11.277948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:11.277894 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:12.142908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.142862 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" event={"ID":"89464194-c498-4655-8868-a930de543542","Type":"ContainerStarted","Data":"91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b"} Oct 30 03:54:12.143324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.142930 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" event={"ID":"89464194-c498-4655-8868-a930de543542","Type":"ContainerStarted","Data":"061cb29f246bb3d53e3f48c4b744d100d272c46a8ca71395af633e375bee4b62"} Oct 30 03:54:12.143324 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.142950 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" event={"ID":"bf49285b-876e-4a6d-8c3e-c8602687306d","Type":"ContainerStarted","Data":"f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1"} Oct 30 03:54:12.253780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.253708 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" podStartSLOduration=2.253692324 podStartE2EDuration="2.253692324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:12.25061881 +0000 UTC m=+3076.252972008" watchObservedRunningTime="2026-10-30 03:54:12.253692324 +0000 UTC m=+3076.256045521" Oct 30 03:54:12.255644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.255607 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" podStartSLOduration=2.255594406 podStartE2EDuration="2.255594406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:12.205132344 +0000 UTC m=+3076.207485544" watchObservedRunningTime="2026-10-30 03:54:12.255594406 +0000 UTC m=+3076.257947611" Oct 30 03:54:12.742741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.741889 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:12.742741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.742135 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 03:54:12.742741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.742211 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 03:54:12.776001 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.775153 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:12.776356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.776333 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 03:54:12.776431 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.776381 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 03:54:12.959245 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:12.959209 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:13.735781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:13.734074 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 03:54:13.735781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:13.734136 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 03:54:13.775059 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:13.775027 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 03:54:13.775219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:13.775079 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 03:54:14.745564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:14.745446 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:14.747994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:14.747328 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 03:54:14.747994 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:14.747413 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 03:54:14.775323 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:14.775279 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 03:54:14.775480 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:14.775351 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 03:54:14.835691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:14.835649 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:15.734159 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:15.734120 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 03:54:15.734467 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:15.734438 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 03:54:15.775842 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:15.775816 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 03:54:15.776279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:15.776245 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 03:54:16.735779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:16.735171 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 03:54:16.735779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:16.735235 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 03:54:16.775191 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:16.775163 4442 patch_prober.go:29] interesting pod/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 03:54:16.775432 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:16.775404 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 03:54:16.800260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:16.800227 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:17.739257 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:17.739220 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:17.739625 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:17.739601 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:17.740358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:17.740343 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 03:54:17.780359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:17.780334 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:17.780627 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:17.780600 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:17.781363 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:17.781345 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 03:54:18.438264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:18.438229 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:19.821506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:19.821478 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:20.923297 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:20.923252 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:21.284328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:21.284249 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:21.386991 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:21.386952 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:21.487352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:21.487319 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:23.159858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:23.159825 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:24.233902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:24.233865 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:54:24.585932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:24.585896 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:24.862245 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:54:24.862163 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:54:26.375671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:26.375641 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:27.997285 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:27.997247 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:29.491364 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:29.491334 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:31.014013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:31.013980 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:32.448808 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:32.448747 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:34.042956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:34.042921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:35.571812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:35.571774 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:37.064910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:37.064879 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:38.502910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:38.502874 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:39.983266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:39.983183 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:41.417887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:41.417845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:43.008350 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:43.008318 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:44.599289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:44.599214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:46.009554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:46.009522 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:47.460827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:47.460789 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:48.871152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:48.871121 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:50.283087 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:50.283060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:50.844618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:50.844576 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:51.784888 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:51.784850 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:51.904805 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:51.902317 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:53.206544 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:53.206509 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:53.650371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:53.650333 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:54.734392 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:54.734359 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:54.891647 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:54:54.891608 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:54:56.208268 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:56.208187 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:57.690878 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:57.690837 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:54:59.182386 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:54:59.182353 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:00.593005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:00.592972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:02.168995 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:02.168962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:04.415732 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:04.415696 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:05.822840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:05.818671 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:06.162256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:06.162163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:07.957188 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:07.957149 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:09.553299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:09.553264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:09.730264 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:09.730227 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:10.064249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:10.064210 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:10.984962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:10.984886 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:11.282816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:11.282783 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:11.501997 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:11.501965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:11.523945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:11.523901 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 03:55:11.524129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:11.523986 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 03:55:13.025414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:13.025381 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:14.852806 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:14.852722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:14.913613 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:14.913579 4442 generic.go:356] "Generic (PLEG): container finished" podID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerID="8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9" exitCode=0 Oct 30 03:55:14.913825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:14.913649 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" event={"ID":"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b","Type":"ContainerDied","Data":"8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9"} Oct 30 03:55:14.914176 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:14.914155 4442 scope.go:122] "RemoveContainer" containerID="8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9" Oct 30 03:55:14.916477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:14.916442 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:55:14.916583 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:14.916516 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:55:14.916583 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:14.916548 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:55:15.924471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:15.924441 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" event={"ID":"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b","Type":"ContainerStarted","Data":"9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d"} Oct 30 03:55:16.517572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:16.517540 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:16.711360 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:16.711322 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:16.813210 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:16.813122 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:55:16.813550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:16.813524 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 03:55:16.813734 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:16.813709 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 03:55:17.811534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:17.811498 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 03:55:17.812046 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:17.811558 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 03:55:18.306315 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.306278 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:18.410182 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.410128 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:18.479389 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.479360 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr"] Oct 30 03:55:18.502649 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.502620 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:18.510393 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.510362 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr"] Oct 30 03:55:18.525657 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.523375 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b267f8\"" Oct 30 03:55:18.599847 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.588203 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b267f8-dockercfg-ggwpk\"" Oct 30 03:55:18.688072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.665209 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:18.688072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.665371 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xcl9\" (UniqueName: \"kubernetes.io/projected/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-kube-api-access-5xcl9\") pod \"payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:18.832318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.831157 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 03:55:18.832318 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.831230 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 03:55:18.832964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.832389 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xcl9\" (UniqueName: \"kubernetes.io/projected/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-kube-api-access-5xcl9\") pod \"payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:18.832964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.832870 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:18.861605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.853607 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:18.907781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:18.904572 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xcl9\" (UniqueName: \"kubernetes.io/projected/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-kube-api-access-5xcl9\") pod \"payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:19.132173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:19.132065 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:19.469740 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:55:19.469700 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38dca31_fcbf_4e8c_babe_a71dfe4fdfc5.slice/crio-eab1b8bfeed341c226ae0359886875aac50a5fb984ce7c2b7ea88975d1cc4087 WatchSource:0}: Error finding container eab1b8bfeed341c226ae0359886875aac50a5fb984ce7c2b7ea88975d1cc4087: Status 404 returned error can't find the container with id eab1b8bfeed341c226ae0359886875aac50a5fb984ce7c2b7ea88975d1cc4087 Oct 30 03:55:19.470740 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:19.470712 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr"] Oct 30 03:55:19.812347 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:19.812309 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 03:55:19.812538 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:19.812362 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 03:55:20.044728 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.044691 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:20.051917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.051886 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" event={"ID":"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5","Type":"ContainerStarted","Data":"a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a"} Oct 30 03:55:20.052071 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.051949 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" event={"ID":"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5","Type":"ContainerStarted","Data":"eab1b8bfeed341c226ae0359886875aac50a5fb984ce7c2b7ea88975d1cc4087"} Oct 30 03:55:20.052130 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.052106 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:20.113656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.113534 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" podStartSLOduration=2.113514285 podStartE2EDuration="2.113514285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:55:20.102943206 +0000 UTC m=+3144.105296404" watchObservedRunningTime="2026-10-30 03:55:20.113514285 +0000 UTC m=+3144.115867483" Oct 30 03:55:20.145652 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.145616 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:20.811372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.811334 4442 patch_prober.go:29] interesting pod/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 03:55:20.811569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:20.811397 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 03:55:21.810218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:21.810184 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:21.817367 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:21.817344 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:55:21.817672 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:21.817651 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:55:21.818325 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:21.818305 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 03:55:23.293500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:23.293445 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:24.788005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:24.787965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:24.909772 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:55:24.909319 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:55:25.031177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:25.031138 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2"] Oct 30 03:55:25.055738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:25.055635 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:25.519379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:25.519347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:26.124372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:26.124340 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:26.418830 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:26.408392 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:27.995462 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:27.995433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:29.661299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:29.661266 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:31.066144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:31.066115 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 03:55:31.149101 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:31.149067 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:31.562058 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:31.562027 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:31.660533 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:31.660498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:31.661289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:31.661269 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:55:32.128779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:32.128119 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:32.918943 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:32.918914 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:34.579828 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:34.579796 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:35.244436 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:35.244399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:36.148937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:36.148900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:37.707941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:37.707903 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:37.822379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:37.822343 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:40.122511 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.122468 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" podUID="9555fbc5-cff4-4090-947e-b11d9bc1b212" containerName="main" containerID="cri-o://0c8572fdb925b4e382a529d5683462dc881af19675ba139256e3af46052a5bf5" gracePeriod=45 Oct 30 03:55:40.138379 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.138353 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:40.418620 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.418581 4442 generic.go:356] "Generic (PLEG): container finished" podID="9555fbc5-cff4-4090-947e-b11d9bc1b212" containerID="0c8572fdb925b4e382a529d5683462dc881af19675ba139256e3af46052a5bf5" exitCode=0 Oct 30 03:55:40.418812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.418626 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" event={"ID":"9555fbc5-cff4-4090-947e-b11d9bc1b212","Type":"ContainerDied","Data":"0c8572fdb925b4e382a529d5683462dc881af19675ba139256e3af46052a5bf5"} Oct 30 03:55:40.418812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.418660 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" event={"ID":"9555fbc5-cff4-4090-947e-b11d9bc1b212","Type":"ContainerDied","Data":"094ee2f4e0ef22ce4514c5ffe3defdb44f70a0b3f4e2a9b861a07b7ace6d110e"} Oct 30 03:55:40.418812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.418675 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094ee2f4e0ef22ce4514c5ffe3defdb44f70a0b3f4e2a9b861a07b7ace6d110e" Oct 30 03:55:40.424742 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.424718 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.504843 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-model-cache\") pod \"9555fbc5-cff4-4090-947e-b11d9bc1b212\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.504949 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9555fbc5-cff4-4090-947e-b11d9bc1b212-kube-api-access-ndrtm\" (UniqueName: \"kubernetes.io/projected/9555fbc5-cff4-4090-947e-b11d9bc1b212-kube-api-access-ndrtm\") pod \"9555fbc5-cff4-4090-947e-b11d9bc1b212\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.505127 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-home\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-home\") pod \"9555fbc5-cff4-4090-947e-b11d9bc1b212\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.505162 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-dshm\") pod \"9555fbc5-cff4-4090-947e-b11d9bc1b212\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.505230 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-model-cache" pod "9555fbc5-cff4-4090-947e-b11d9bc1b212" (UID: "9555fbc5-cff4-4090-947e-b11d9bc1b212"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.505426 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9555fbc5-cff4-4090-947e-b11d9bc1b212-tls-certs\" (UniqueName: \"kubernetes.io/secret/9555fbc5-cff4-4090-947e-b11d9bc1b212-tls-certs\") pod \"9555fbc5-cff4-4090-947e-b11d9bc1b212\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.505462 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-tmp-dir\") pod \"9555fbc5-cff4-4090-947e-b11d9bc1b212\" (UID: \"9555fbc5-cff4-4090-947e-b11d9bc1b212\") " Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.506151 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:55:40.506646 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.506561 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-home" pod "9555fbc5-cff4-4090-947e-b11d9bc1b212" (UID: "9555fbc5-cff4-4090-947e-b11d9bc1b212"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:40.507442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.507408 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-tmp-dir" pod "9555fbc5-cff4-4090-947e-b11d9bc1b212" (UID: "9555fbc5-cff4-4090-947e-b11d9bc1b212"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:40.508073 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.508052 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-dshm" pod "9555fbc5-cff4-4090-947e-b11d9bc1b212" (UID: "9555fbc5-cff4-4090-947e-b11d9bc1b212"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:55:40.508306 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.508286 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9555fbc5-cff4-4090-947e-b11d9bc1b212-kube-api-access-ndrtm" pod "9555fbc5-cff4-4090-947e-b11d9bc1b212" (UID: "9555fbc5-cff4-4090-947e-b11d9bc1b212"). InnerVolumeSpecName "kube-api-access-ndrtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:55:40.508737 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.508718 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9555fbc5-cff4-4090-947e-b11d9bc1b212-tls-certs" pod "9555fbc5-cff4-4090-947e-b11d9bc1b212" (UID: "9555fbc5-cff4-4090-947e-b11d9bc1b212"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:55:40.579920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.579891 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:40.607231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.607205 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:55:40.607231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.607227 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:55:40.607231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.607237 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9555fbc5-cff4-4090-947e-b11d9bc1b212-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:55:40.607443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.607245 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9555fbc5-cff4-4090-947e-b11d9bc1b212-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:55:40.607443 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:40.607254 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrtm\" (UniqueName: \"kubernetes.io/projected/9555fbc5-cff4-4090-947e-b11d9bc1b212-kube-api-access-ndrtm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:55:41.212819 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:41.212787 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:41.469858 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:41.468478 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2" Oct 30 03:55:41.586780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:41.585861 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2"] Oct 30 03:55:41.594898 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:41.594858 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e84013/test-model-e84013-kserve-8575bdfd9c-vzbj2"] Oct 30 03:55:41.932803 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:41.932774 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:42.200651 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:42.200562 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9555fbc5-cff4-4090-947e-b11d9bc1b212" path="/var/lib/kubelet/pods/9555fbc5-cff4-4090-947e-b11d9bc1b212/volumes" Oct 30 03:55:42.904889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:42.904856 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:43.615499 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:43.615469 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:43.717840 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:43.717803 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:44.305459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:44.305423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:44.453607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:44.453573 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:44.685549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:44.685475 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:46.088671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:46.088632 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:46.194197 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:46.193831 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:46.481876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:46.481782 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:47.679608 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:47.679574 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:49.075213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:49.075181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:50.602078 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:50.602043 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:50.963321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:50.963221 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:51.711862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:51.711827 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:51.804006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:51.803974 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:52.002283 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:52.002246 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:52.197475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:52.197439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:53.024844 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:53.024814 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:53.759300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:53.759264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:53.923681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:53.923650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:54.720184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:54.720152 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh"] Oct 30 03:55:54.720570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:54.720368 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" containerID="cri-o://677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8" gracePeriod=30 Oct 30 03:55:54.732951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:54.732924 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh"] Oct 30 03:55:54.733172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:54.733138 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" containerID="cri-o://3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e" gracePeriod=30 Oct 30 03:55:54.737554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:54.737531 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl"] Oct 30 03:55:54.737823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:54.737796 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" containerID="cri-o://25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c" gracePeriod=30 Oct 30 03:55:54.872314 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:55:54.872278 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:55:55.494854 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:55.494822 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:57.096790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:57.096714 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:58.073382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:58.073344 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q"] Oct 30 03:55:58.102080 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:58.102045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:58.549466 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:58.549436 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:58.683430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:58.683397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:55:59.046006 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:55:59.045971 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:00.063095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.062122 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:56:00.114910 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.114877 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:56:00.124948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.124914 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.153925 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-socket\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154047 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-envoy\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154088 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-podinfo\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154134 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-credential-socket\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-data\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154514 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-socket" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154543 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-certs\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154681 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istiod-ca-cert\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.154729 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-token\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-token\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.156504 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-kube-api-access-mdfvv\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-kube-api-access-mdfvv\") pod \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\" (UID: \"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf\") " Oct 30 03:56:00.158061 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.157515 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.158882 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.158694 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-data" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.158942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.158916 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-credential-socket" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.159072 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.159041 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-certs" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.159251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.159230 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istiod-ca-cert" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:00.164288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.164246 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-podinfo" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:00.165098 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.165056 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-token" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:00.165198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.165178 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-kube-api-access-mdfvv" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "kube-api-access-mdfvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:00.165299 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.165280 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-envoy" pod "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" (UID: "e04be625-d5ba-40b8-a4b2-cb4b80a0afaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.258723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.258686 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istiod-ca-cert\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.258926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.258739 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-certs\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.258926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.258878 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-envoy\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.259053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.258977 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-kube-api-access-mjjbw\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-kube-api-access-mjjbw\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259023 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-data\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259030 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-certs" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.259225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259052 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-credential-socket\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.259225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259084 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-podinfo\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.259225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259174 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-istio-token\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-istio-token\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259210 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istiod-ca-cert" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:00.259451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259238 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-credential-socket\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259273 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-credential-socket" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.259451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259281 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-socket\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259360 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-certs\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259392 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-token\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-token\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.259902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259524 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-socket" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.259902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259534 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb8efd97-9c40-438c-9943-12236f7ac198-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8efd97-9c40-438c-9943-12236f7ac198-istio-podinfo\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259574 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-credential-socket" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.259902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259662 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb8efd97-9c40-438c-9943-12236f7ac198-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8efd97-9c40-438c-9943-12236f7ac198-istiod-ca-cert\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.259902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259712 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-kube-api-access-czjzc\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-kube-api-access-czjzc\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.259902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259793 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-data\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.259902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259770 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-certs" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.260307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259891 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-envoy\") pod \"eb8efd97-9c40-438c-9943-12236f7ac198\" (UID: \"eb8efd97-9c40-438c-9943-12236f7ac198\") " Oct 30 03:56:00.260307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.259958 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-socket\") pod \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\" (UID: \"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b\") " Oct 30 03:56:00.260430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.260323 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-data" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.260534 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.260511 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-socket" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.261019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.260984 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb8efd97-9c40-438c-9943-12236f7ac198-istiod-ca-cert" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:00.261258 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261232 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261280 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261300 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261314 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261326 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261339 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261353 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb8efd97-9c40-438c-9943-12236f7ac198-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261365 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261377 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261390 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261403 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261415 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261428 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261440 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261453 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261465 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.261532 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.261476 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfvv\" (UniqueName: \"kubernetes.io/projected/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf-kube-api-access-mdfvv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.262846 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.262817 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-envoy" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.262944 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.262870 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-istio-token" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:00.263005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.262969 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-data" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.263144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.263124 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-podinfo" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:00.263184 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.263136 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-envoy" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:00.263329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.263309 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-kube-api-access-mjjbw" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "kube-api-access-mjjbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:00.263607 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.263586 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb8efd97-9c40-438c-9943-12236f7ac198-istio-podinfo" pod "eb8efd97-9c40-438c-9943-12236f7ac198" (UID: "eb8efd97-9c40-438c-9943-12236f7ac198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:00.264158 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.264140 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-token" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:00.264506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.264490 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-kube-api-access-czjzc" pod "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" (UID: "a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b"). InnerVolumeSpecName "kube-api-access-czjzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363839 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363864 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363872 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb8efd97-9c40-438c-9943-12236f7ac198-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363881 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czjzc\" (UniqueName: \"kubernetes.io/projected/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-kube-api-access-czjzc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363895 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363903 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb8efd97-9c40-438c-9943-12236f7ac198-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363910 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.363915 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363923 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjjbw\" (UniqueName: \"kubernetes.io/projected/eb8efd97-9c40-438c-9943-12236f7ac198-kube-api-access-mjjbw\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.364304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.363931 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:00.387180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.387147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:00.731427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.731342 4442 generic.go:356] "Generic (PLEG): container finished" podID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerID="677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8" exitCode=0 Oct 30 03:56:00.731427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.731411 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" Oct 30 03:56:00.731427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.731419 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" event={"ID":"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf","Type":"ContainerDied","Data":"677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8"} Oct 30 03:56:00.732225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.731472 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh" event={"ID":"e04be625-d5ba-40b8-a4b2-cb4b80a0afaf","Type":"ContainerDied","Data":"579db6c28c0ae17b5dfe4f2e0bfed616e646f9cd4741d7a2681e042f0b181877"} Oct 30 03:56:00.732225 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.731501 4442 scope.go:122] "RemoveContainer" containerID="677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8" Oct 30 03:56:00.733295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.733264 4442 generic.go:356] "Generic (PLEG): container finished" podID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerID="25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c" exitCode=0 Oct 30 03:56:00.733434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.733327 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" Oct 30 03:56:00.733434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.733364 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" event={"ID":"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b","Type":"ContainerDied","Data":"25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c"} Oct 30 03:56:00.733434 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.733416 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl" event={"ID":"a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b","Type":"ContainerDied","Data":"e3c969f418fd792566832711c92562dcd4469aaffdd79832d43bd221205aa23f"} Oct 30 03:56:00.735280 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.735255 4442 generic.go:356] "Generic (PLEG): container finished" podID="eb8efd97-9c40-438c-9943-12236f7ac198" containerID="3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e" exitCode=0 Oct 30 03:56:00.735391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.735307 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" event={"ID":"eb8efd97-9c40-438c-9943-12236f7ac198","Type":"ContainerDied","Data":"3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e"} Oct 30 03:56:00.735391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.735322 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" Oct 30 03:56:00.735391 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.735344 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh" event={"ID":"eb8efd97-9c40-438c-9943-12236f7ac198","Type":"ContainerDied","Data":"7f078d5baf4272fe60bf3dc4b58533062e75498f5223380e704796fb9f86c9fa"} Oct 30 03:56:00.760707 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.760680 4442 scope.go:122] "RemoveContainer" containerID="677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8" Oct 30 03:56:00.761326 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:00.761288 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8\": container with ID starting with 677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8 not found: ID does not exist" containerID="677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8" Oct 30 03:56:00.761445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.761336 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8"} err="failed to get container status \"677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8\": rpc error: code = NotFound desc = could not find container \"677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8\": container with ID starting with 677842b1eb02f95983a7da6a19260b71dac728650a445e52a48c7f6b340a80f8 not found: ID does not exist" Oct 30 03:56:00.761445 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.761360 4442 scope.go:122] "RemoveContainer" containerID="25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c" Oct 30 03:56:00.784524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.784504 4442 scope.go:122] "RemoveContainer" containerID="25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c" Oct 30 03:56:00.784901 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:00.784878 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c\": container with ID starting with 25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c not found: ID does not exist" containerID="25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c" Oct 30 03:56:00.784962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.784911 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c"} err="failed to get container status \"25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c\": rpc error: code = NotFound desc = could not find container \"25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c\": container with ID starting with 25c6696495843f32690dcde3a547c664acaebfe2b7557500c852634271639f9c not found: ID does not exist" Oct 30 03:56:00.784962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.784929 4442 scope.go:122] "RemoveContainer" containerID="3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e" Oct 30 03:56:00.795937 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.795916 4442 scope.go:122] "RemoveContainer" containerID="3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e" Oct 30 03:56:00.796257 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:00.796236 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e\": container with ID starting with 3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e not found: ID does not exist" containerID="3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e" Oct 30 03:56:00.796337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.796266 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e"} err="failed to get container status \"3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e\": rpc error: code = NotFound desc = could not find container \"3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e\": container with ID starting with 3fdaeb4c6aac41a80d614be378b307db2abf41a435e97fdf8364633d882a381e not found: ID does not exist" Oct 30 03:56:00.819896 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.819867 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh"] Oct 30 03:56:00.830574 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.830551 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-9p4jh"] Oct 30 03:56:00.853085 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.853049 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl"] Oct 30 03:56:00.856416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.856396 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-tvsnl"] Oct 30 03:56:00.888731 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.888707 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh"] Oct 30 03:56:00.892213 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:00.892186 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-wgghh"] Oct 30 03:56:01.792524 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:01.792492 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:01.928799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:01.928703 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z"] Oct 30 03:56:01.929017 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:01.928994 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" containerID="cri-o://f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f" gracePeriod=30 Oct 30 03:56:02.109476 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:02.109397 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" path="/var/lib/kubelet/pods/a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b/volumes" Oct 30 03:56:02.109941 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:02.109927 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" path="/var/lib/kubelet/pods/e04be625-d5ba-40b8-a4b2-cb4b80a0afaf/volumes" Oct 30 03:56:02.110440 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:02.110424 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" path="/var/lib/kubelet/pods/eb8efd97-9c40-438c-9943-12236f7ac198/volumes" Oct 30 03:56:03.283661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:03.283629 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:04.373614 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:04.373581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:04.472198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:04.472163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:04.871970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:04.871934 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:05.087827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:05.080165 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:06.485602 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:06.485569 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:07.218470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.218410 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:56:07.393056 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393017 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-data\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.393274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393081 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-kube-api-access-zhxwf\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-kube-api-access-zhxwf\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.393274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393124 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-certs\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.393274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393156 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-socket\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.393470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393344 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-podinfo\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.393470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393381 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-token\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-token\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.393470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393409 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d390a8d-b124-4c7d-8097-aaf88b393175-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d390a8d-b124-4c7d-8097-aaf88b393175-istiod-ca-cert\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.393650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393474 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-data" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:07.393650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393508 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-certs" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:07.393650 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393618 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-socket" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:07.394186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393681 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-credential-socket\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.394314 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.394218 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-envoy\") pod \"4d390a8d-b124-4c7d-8097-aaf88b393175\" (UID: \"4d390a8d-b124-4c7d-8097-aaf88b393175\") " Oct 30 03:56:07.394839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393956 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-credential-socket" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:07.394839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.393986 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d390a8d-b124-4c7d-8097-aaf88b393175-istiod-ca-cert" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:07.395023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.394966 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d390a8d-b124-4c7d-8097-aaf88b393175-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.395023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.394990 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.395023 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.395008 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.395169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.395022 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.395169 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.395125 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.396198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.396124 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-token" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:07.396198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.396161 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-podinfo" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:07.396658 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.396637 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-envoy" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:07.397353 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.397334 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-kube-api-access-zhxwf" pod "4d390a8d-b124-4c7d-8097-aaf88b393175" (UID: "4d390a8d-b124-4c7d-8097-aaf88b393175"). InnerVolumeSpecName "kube-api-access-zhxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:07.496801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.496703 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.496801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.496733 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhxwf\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-kube-api-access-zhxwf\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.496801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.496745 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.496801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.496789 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d390a8d-b124-4c7d-8097-aaf88b393175-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:07.792948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.792909 4442 generic.go:356] "Generic (PLEG): container finished" podID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerID="f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f" exitCode=0 Oct 30 03:56:07.793140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.792993 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" Oct 30 03:56:07.793140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.793026 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" event={"ID":"4d390a8d-b124-4c7d-8097-aaf88b393175","Type":"ContainerDied","Data":"f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f"} Oct 30 03:56:07.793140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.793088 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z" event={"ID":"4d390a8d-b124-4c7d-8097-aaf88b393175","Type":"ContainerDied","Data":"ecf3514e25dd9fd570046a0b7bcf7a3c10b7006266a12ed36366e4a964fa721b"} Oct 30 03:56:07.793140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.793118 4442 scope.go:122] "RemoveContainer" containerID="f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f" Oct 30 03:56:07.805510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.805489 4442 scope.go:122] "RemoveContainer" containerID="f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f" Oct 30 03:56:07.805795 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:07.805768 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f\": container with ID starting with f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f not found: ID does not exist" containerID="f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f" Oct 30 03:56:07.805894 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.805801 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f"} err="failed to get container status \"f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f\": rpc error: code = NotFound desc = could not find container \"f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f\": container with ID starting with f4f8baac165b679c39a2123213cfb7c581cb4ab0bc9d6750f5499e467a22de3f not found: ID does not exist" Oct 30 03:56:07.853150 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.853112 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z"] Oct 30 03:56:07.864481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.864457 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-7sb5z"] Oct 30 03:56:07.909441 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.909415 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk"] Oct 30 03:56:07.910221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910205 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910222 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910237 4442 cpu_manager.go:414] "containerMap: removing container" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910242 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910259 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910265 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910273 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910277 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910285 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9555fbc5-cff4-4090-947e-b11d9bc1b212" containerName="main" Oct 30 03:56:07.910319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910289 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9555fbc5-cff4-4090-947e-b11d9bc1b212" containerName="main" Oct 30 03:56:07.910632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910403 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79de6e0-ad3d-4cd7-9bde-8cdc5633ed8b" containerName="istio-proxy" Oct 30 03:56:07.910632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910416 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb8efd97-9c40-438c-9943-12236f7ac198" containerName="istio-proxy" Oct 30 03:56:07.910632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910426 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04be625-d5ba-40b8-a4b2-cb4b80a0afaf" containerName="istio-proxy" Oct 30 03:56:07.910632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910431 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" containerName="istio-proxy" Oct 30 03:56:07.910632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.910436 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9555fbc5-cff4-4090-947e-b11d9bc1b212" containerName="main" Oct 30 03:56:07.915632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.915616 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:07.928946 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.928918 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b267f8-serving-cert\"" Oct 30 03:56:07.933561 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.933535 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk"] Oct 30 03:56:07.970356 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:07.970325 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:08.047470 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.047396 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:08.104832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.104803 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-metrics-certs\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.105013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.104933 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.105077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.105020 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-maas-api-tls\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.105249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.105228 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzg4t\" (UniqueName: \"kubernetes.io/projected/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-kube-api-access-vzg4t\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.109248 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.109224 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d390a8d-b124-4c7d-8097-aaf88b393175" path="/var/lib/kubelet/pods/4d390a8d-b124-4c7d-8097-aaf88b393175/volumes" Oct 30 03:56:08.148097 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.148069 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:08.214145 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.214103 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzg4t\" (UniqueName: \"kubernetes.io/projected/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-kube-api-access-vzg4t\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.214897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.214869 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-metrics-certs\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.215219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.215197 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.215412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.215393 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-maas-api-tls\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.221478 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.221440 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.253982 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.253944 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-maas-api-tls\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.257008 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.256413 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-metrics-certs\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.265901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.265868 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzg4t\" (UniqueName: \"kubernetes.io/projected/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-kube-api-access-vzg4t\") pod \"maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.530365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.530329 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:08.837296 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:56:08.837268 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d210c9_e8b4_4cf0_a5db_a1b0cbf8382a.slice/crio-81e8b933358cd064fa46de4cf639d66c2eee836036c6d83a4f5c8e79545afbbb WatchSource:0}: Error finding container 81e8b933358cd064fa46de4cf639d66c2eee836036c6d83a4f5c8e79545afbbb: Status 404 returned error can't find the container with id 81e8b933358cd064fa46de4cf639d66c2eee836036c6d83a4f5c8e79545afbbb Oct 30 03:56:08.837503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.837480 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk"] Oct 30 03:56:08.838837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:08.838818 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:56:09.483945 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:09.483904 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:09.812183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:09.812145 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" event={"ID":"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a","Type":"ContainerStarted","Data":"58301e0fad69ada642970eee7baac3900968889279691fac4488f9e4e37b3841"} Oct 30 03:56:09.812629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:09.812230 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" event={"ID":"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a","Type":"ContainerStarted","Data":"81e8b933358cd064fa46de4cf639d66c2eee836036c6d83a4f5c8e79545afbbb"} Oct 30 03:56:09.812629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:09.812254 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:09.847419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:09.847365 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" podStartSLOduration=2.438592706 podStartE2EDuration="2.847349077s" totalImagesPullingTime="408.756371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:09.846821075 +0000 UTC m=+3193.849174269" watchObservedRunningTime="2026-10-30 03:56:09.847349077 +0000 UTC m=+3193.849702275" Oct 30 03:56:10.970963 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:10.970931 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:11.163948 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:11.163915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:12.499129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:12.499098 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:13.102118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.102075 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" podUID="174ed8a3-1a7f-45b0-816f-b5fd461b6a24" containerName="main" containerID="cri-o://9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41" gracePeriod=45 Oct 30 03:56:13.399510 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.399488 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:56:13.477683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.477645 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tmp-dir\") pod \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " Oct 30 03:56:13.477907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.477790 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-kube-api-access-2zfk2\" (UniqueName: \"kubernetes.io/projected/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-kube-api-access-2zfk2\") pod \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " Oct 30 03:56:13.477907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.477823 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-model-cache\") pod \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " Oct 30 03:56:13.477907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.477899 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tls-certs\" (UniqueName: \"kubernetes.io/secret/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tls-certs\") pod \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " Oct 30 03:56:13.478103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.477927 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-dshm\") pod \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " Oct 30 03:56:13.478103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.477962 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tmp-dir" pod "174ed8a3-1a7f-45b0-816f-b5fd461b6a24" (UID: "174ed8a3-1a7f-45b0-816f-b5fd461b6a24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:13.478103 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.478089 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-home\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-home\") pod \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\" (UID: \"174ed8a3-1a7f-45b0-816f-b5fd461b6a24\") " Oct 30 03:56:13.478274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.478130 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-model-cache" pod "174ed8a3-1a7f-45b0-816f-b5fd461b6a24" (UID: "174ed8a3-1a7f-45b0-816f-b5fd461b6a24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:13.478400 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.478377 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-home" pod "174ed8a3-1a7f-45b0-816f-b5fd461b6a24" (UID: "174ed8a3-1a7f-45b0-816f-b5fd461b6a24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:13.479063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.478978 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.479063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.479012 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.479063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.479027 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.480942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.480908 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tls-certs" pod "174ed8a3-1a7f-45b0-816f-b5fd461b6a24" (UID: "174ed8a3-1a7f-45b0-816f-b5fd461b6a24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:13.480942 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.480918 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-dshm" pod "174ed8a3-1a7f-45b0-816f-b5fd461b6a24" (UID: "174ed8a3-1a7f-45b0-816f-b5fd461b6a24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:13.481547 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.481519 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-kube-api-access-2zfk2" pod "174ed8a3-1a7f-45b0-816f-b5fd461b6a24" (UID: "174ed8a3-1a7f-45b0-816f-b5fd461b6a24"). InnerVolumeSpecName "kube-api-access-2zfk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:13.580414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.580384 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zfk2\" (UniqueName: \"kubernetes.io/projected/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-kube-api-access-2zfk2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.580414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.580411 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.580810 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.580421 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/174ed8a3-1a7f-45b0-816f-b5fd461b6a24-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:13.847416 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.847382 4442 generic.go:356] "Generic (PLEG): container finished" podID="174ed8a3-1a7f-45b0-816f-b5fd461b6a24" containerID="9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41" exitCode=0 Oct 30 03:56:13.847629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.847423 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" Oct 30 03:56:13.847629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.847447 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" event={"ID":"174ed8a3-1a7f-45b0-816f-b5fd461b6a24","Type":"ContainerDied","Data":"9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41"} Oct 30 03:56:13.847629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.847490 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q" event={"ID":"174ed8a3-1a7f-45b0-816f-b5fd461b6a24","Type":"ContainerDied","Data":"68409f8e24eb77943a15ea37fd47fd192b6c2b45805f26b555badf3611639e31"} Oct 30 03:56:13.847629 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.847511 4442 scope.go:122] "RemoveContainer" containerID="9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41" Oct 30 03:56:13.859365 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.859349 4442 scope.go:122] "RemoveContainer" containerID="9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41" Oct 30 03:56:13.859630 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:13.859607 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41\": container with ID starting with 9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41 not found: ID does not exist" containerID="9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41" Oct 30 03:56:13.859715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.859637 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41"} err="failed to get container status \"9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41\": rpc error: code = NotFound desc = could not find container \"9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41\": container with ID starting with 9f7d7ccd77014753baa44d935322f9427bb7eb0bf4977d8df05d00198158ce41 not found: ID does not exist" Oct 30 03:56:13.908585 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.908513 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q"] Oct 30 03:56:13.946681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:13.946627 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-35fc87/test-model-35fc87-kserve-b78678587-dqk2q"] Oct 30 03:56:14.145329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:14.139441 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="174ed8a3-1a7f-45b0-816f-b5fd461b6a24" path="/var/lib/kubelet/pods/174ed8a3-1a7f-45b0-816f-b5fd461b6a24/volumes" Oct 30 03:56:14.152063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:14.152031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:15.827149 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:15.827122 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 03:56:15.953122 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:15.953089 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:17.577972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:17.577936 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:19.097606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:19.097564 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:20.547422 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:20.547387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:22.019284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:22.019252 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:23.489063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:23.489030 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:24.862474 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:24.862435 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:56:24.881821 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:24.881786 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:26.276485 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:26.276454 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:27.809464 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:27.809434 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:29.272668 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:29.272618 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:30.681902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:30.681867 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:32.211736 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:32.211704 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:33.665747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:33.665717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:35.061181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:35.061148 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:36.504303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:36.504269 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:37.901009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:37.900969 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:39.126260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:39.126225 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:39.932195 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:39.932160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:41.121928 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:41.121900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:41.224609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:41.224573 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:42.188743 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:42.188691 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:42.297899 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:42.297866 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:42.575446 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:42.575412 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp"] Oct 30 03:56:42.575700 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:42.575669 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" containerID="cri-o://c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a" gracePeriod=30 Oct 30 03:56:42.584251 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:42.584218 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm"] Oct 30 03:56:42.584471 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:42.584449 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" containerID="cri-o://f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06" gracePeriod=30 Oct 30 03:56:42.592219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:42.592192 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:43.476908 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:43.476854 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:43.581054 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:43.581023 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:43.677410 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:43.677355 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:43.782349 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:43.782277 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:43.915792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:43.915727 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:44.275475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:44.275439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:44.637004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:44.636920 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8"] Oct 30 03:56:44.637447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:44.637286 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" containerID="cri-o://5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129" gracePeriod=30 Oct 30 03:56:44.651689 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:44.651653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:45.438841 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:45.438810 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:45.788686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:45.788652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:47.284920 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.284890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:47.880123 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.880097 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:56:47.906640 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.906614 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:56:47.920983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.920957 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-envoy\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921029 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-kube-api-access-vq5vr\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-kube-api-access-vq5vr\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921194 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921123 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istiod-ca-cert\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921199 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-certs\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921260 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921234 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-token\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-token\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921267 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-data\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921372 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921295 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-credential-socket\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921394 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-podinfo\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921482 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921440 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-socket\") pod \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\" (UID: \"9f78fc12-1193-44e0-ad47-d0db6a795b3a\") " Oct 30 03:56:47.921605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921525 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-certs" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.921605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921541 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-data" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.921720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921666 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istiod-ca-cert" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:47.921720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921679 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-credential-socket" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.922407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.921825 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-socket" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.922407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.922252 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.922407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.922269 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.922407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.922277 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.922407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.922286 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.922407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.922298 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:47.924421 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.924397 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-envoy" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:47.924508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.924463 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-token" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:47.924558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.924524 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-kube-api-access-vq5vr" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "kube-api-access-vq5vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:47.925579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:47.925553 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-podinfo" pod "9f78fc12-1193-44e0-ad47-d0db6a795b3a" (UID: "9f78fc12-1193-44e0-ad47-d0db6a795b3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:48.023705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.023668 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-kube-api-access-sv8kq\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-kube-api-access-sv8kq\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.024675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.024616 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-data" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.023741 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-data\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.026093 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-envoy\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.026146 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-token\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-token\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.026180 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istiod-ca-cert\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.026324 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-credential-socket\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.026444 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-socket\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.026483 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-certs\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.026641 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.026519 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-podinfo\") pod \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\" (UID: \"9331b8fa-1b36-48c8-bb92-49ea25d8e66b\") " Oct 30 03:56:48.028509 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.028465 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-certs" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:48.028956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.028396 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-socket" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:48.028956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.028811 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-credential-socket" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:48.028956 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.028946 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istiod-ca-cert" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:48.029465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029438 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029470 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029486 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029500 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029514 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029527 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029541 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq5vr\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-kube-api-access-vq5vr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029555 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.029569 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.029569 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f78fc12-1193-44e0-ad47-d0db6a795b3a-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.031972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.031944 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-podinfo" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:48.032496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.032363 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-kube-api-access-sv8kq" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "kube-api-access-sv8kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:48.033786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.033744 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-token" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:48.037090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.037061 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-envoy" pod "9331b8fa-1b36-48c8-bb92-49ea25d8e66b" (UID: "9331b8fa-1b36-48c8-bb92-49ea25d8e66b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:48.114648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.114563 4442 generic.go:356] "Generic (PLEG): container finished" podID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerID="f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06" exitCode=0 Oct 30 03:56:48.114863 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.114679 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" Oct 30 03:56:48.115224 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.115194 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" event={"ID":"9331b8fa-1b36-48c8-bb92-49ea25d8e66b","Type":"ContainerDied","Data":"f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06"} Oct 30 03:56:48.115371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.115249 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm" event={"ID":"9331b8fa-1b36-48c8-bb92-49ea25d8e66b","Type":"ContainerDied","Data":"69955593565b6bafb4deae1b225e7fcd3837b08d1bd5014c326dd12f564d2a95"} Oct 30 03:56:48.115371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.115283 4442 scope.go:122] "RemoveContainer" containerID="f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06" Oct 30 03:56:48.116475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.116340 4442 generic.go:356] "Generic (PLEG): container finished" podID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerID="c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a" exitCode=0 Oct 30 03:56:48.116475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.116385 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" Oct 30 03:56:48.116475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.116406 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" event={"ID":"9f78fc12-1193-44e0-ad47-d0db6a795b3a","Type":"ContainerDied","Data":"c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a"} Oct 30 03:56:48.116475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.116443 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp" event={"ID":"9f78fc12-1193-44e0-ad47-d0db6a795b3a","Type":"ContainerDied","Data":"14af6dab89686b809bfa2ebc63c4e9356392cc59c96a64a64e9e08fc92d92e23"} Oct 30 03:56:48.129069 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.129052 4442 scope.go:122] "RemoveContainer" containerID="f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06" Oct 30 03:56:48.129350 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:48.129330 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06\": container with ID starting with f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06 not found: ID does not exist" containerID="f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06" Oct 30 03:56:48.129459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.129356 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06"} err="failed to get container status \"f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06\": rpc error: code = NotFound desc = could not find container \"f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06\": container with ID starting with f2ed8c89f3ac555eda0ab9ef91a722c8bd57c42928b73547e1179733591e0d06 not found: ID does not exist" Oct 30 03:56:48.129459 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.129374 4442 scope.go:122] "RemoveContainer" containerID="c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a" Oct 30 03:56:48.131009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.130880 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv8kq\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-kube-api-access-sv8kq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.131009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.130905 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.131009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.130920 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.131009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.130933 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331b8fa-1b36-48c8-bb92-49ea25d8e66b-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:48.139591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.139572 4442 scope.go:122] "RemoveContainer" containerID="c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a" Oct 30 03:56:48.139897 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:48.139874 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a\": container with ID starting with c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a not found: ID does not exist" containerID="c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a" Oct 30 03:56:48.139974 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.139903 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a"} err="failed to get container status \"c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a\": rpc error: code = NotFound desc = could not find container \"c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a\": container with ID starting with c1b98bb1aedb5d8498ca88653f505612e0a30c3c1dbea8c0de75a89295e3a32a not found: ID does not exist" Oct 30 03:56:48.176144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.176112 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp"] Oct 30 03:56:48.179777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.179731 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-pzwsp"] Oct 30 03:56:48.210581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.210537 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm"] Oct 30 03:56:48.214341 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.214306 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e84013-openshift-default-86489877f9-vhdtm"] Oct 30 03:56:48.787186 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:48.787152 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:49.924881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:49.924823 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:56:50.057327 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057292 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-data\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057504 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057432 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-socket\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057554 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057502 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-kube-api-access-4dh4p\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-kube-api-access-4dh4p\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057605 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057590 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-certs\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057643 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057619 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-data" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:50.057685 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057665 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-envoy\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057738 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-credential-socket\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057813 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfbb56a-d834-41fa-b559-09e8be196c7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbb56a-d834-41fa-b559-09e8be196c7e-istiod-ca-cert\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057873 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057861 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-podinfo\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.057971 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.057955 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-token\") pod \"8cfbb56a-d834-41fa-b559-09e8be196c7e\" (UID: \"8cfbb56a-d834-41fa-b559-09e8be196c7e\") " Oct 30 03:56:50.058412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.058155 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-credential-socket" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:50.058412 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.058371 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-certs" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:50.058683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.058481 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfbb56a-d834-41fa-b559-09e8be196c7e-istiod-ca-cert" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:50.058683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.058675 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.058855 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.058697 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.058855 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.058710 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cfbb56a-d834-41fa-b559-09e8be196c7e-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.058855 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.058724 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.060687 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.060656 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-envoy" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:50.061025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.061007 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-kube-api-access-4dh4p" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "kube-api-access-4dh4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:50.061326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.061304 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-socket" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:56:50.061378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.061333 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-token" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:50.062375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.062355 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-podinfo" pod "8cfbb56a-d834-41fa-b559-09e8be196c7e" (UID: "8cfbb56a-d834-41fa-b559-09e8be196c7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:56:50.109291 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.109264 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" path="/var/lib/kubelet/pods/9331b8fa-1b36-48c8-bb92-49ea25d8e66b/volumes" Oct 30 03:56:50.109780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.109766 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" path="/var/lib/kubelet/pods/9f78fc12-1193-44e0-ad47-d0db6a795b3a/volumes" Oct 30 03:56:50.151764 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.151729 4442 generic.go:356] "Generic (PLEG): container finished" podID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerID="5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129" exitCode=0 Oct 30 03:56:50.151903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.151788 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" event={"ID":"8cfbb56a-d834-41fa-b559-09e8be196c7e","Type":"ContainerDied","Data":"5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129"} Oct 30 03:56:50.151903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.151843 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" event={"ID":"8cfbb56a-d834-41fa-b559-09e8be196c7e","Type":"ContainerDied","Data":"64b869aabfc90d33409a6a94d533983f52d0d911b0e6dd8a2c7769a22bcad707"} Oct 30 03:56:50.151903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.151863 4442 scope.go:122] "RemoveContainer" containerID="5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129" Oct 30 03:56:50.151903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.151797 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8" Oct 30 03:56:50.161133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.161037 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.161133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.161065 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.161133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.161080 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.161133 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.161094 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cfbb56a-d834-41fa-b559-09e8be196c7e-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.161473 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.161108 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dh4p\" (UniqueName: \"kubernetes.io/projected/8cfbb56a-d834-41fa-b559-09e8be196c7e-kube-api-access-4dh4p\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:50.163796 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.163736 4442 scope.go:122] "RemoveContainer" containerID="5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129" Oct 30 03:56:50.164078 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:50.164061 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129\": container with ID starting with 5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129 not found: ID does not exist" containerID="5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129" Oct 30 03:56:50.164136 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.164085 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129"} err="failed to get container status \"5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129\": rpc error: code = NotFound desc = could not find container \"5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129\": container with ID starting with 5fc40e563312c6cc62560e524c26e8d9dd2afef451ce31783a41ff54a93db129 not found: ID does not exist" Oct 30 03:56:50.201874 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.201793 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8"] Oct 30 03:56:50.226738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.208219 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d53363-openshift-default-5568f6bb59-r7xk8"] Oct 30 03:56:50.280414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:50.280385 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:51.096966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:51.095775 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:51.101469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:51.101434 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:56:51.777310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:51.777259 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:51.889419 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:51.889337 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:52.081951 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:52.081864 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529"] Oct 30 03:56:52.109616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:52.109588 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" path="/var/lib/kubelet/pods/8cfbb56a-d834-41fa-b559-09e8be196c7e/volumes" Oct 30 03:56:52.132371 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:52.132347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:52.233019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:52.232984 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:52.333792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:52.333635 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:53.585003 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:53.584971 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:53.906792 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:53.906688 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:54.862795 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:56:54.862736 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:56:55.143811 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:55.143722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:56.530955 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:56.530920 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:57.083200 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:57.083151 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" podUID="5e77fc9f-bada-453e-8b68-b57512b8850b" containerName="payload-processing" containerID="cri-o://3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579" gracePeriod=25 Oct 30 03:56:57.624926 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:57.624847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:57.701180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:57.701146 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62"] Oct 30 03:56:57.701448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:57.701417 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" podUID="f2c55a01-06ca-407d-9f40-bcd70007c00a" containerName="maas-api" containerID="cri-o://fa287b11dc833270445ec787ade2d2a3a695513ca56574bd8167e82312b5f4ed" gracePeriod=30 Oct 30 03:56:57.747451 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:57.747246 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:58.315289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.305388 4442 generic.go:356] "Generic (PLEG): container finished" podID="f2c55a01-06ca-407d-9f40-bcd70007c00a" containerID="fa287b11dc833270445ec787ade2d2a3a695513ca56574bd8167e82312b5f4ed" exitCode=0 Oct 30 03:56:58.346786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.336769 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" event={"ID":"f2c55a01-06ca-407d-9f40-bcd70007c00a","Type":"ContainerDied","Data":"fa287b11dc833270445ec787ade2d2a3a695513ca56574bd8167e82312b5f4ed"} Oct 30 03:56:58.346786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.336849 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" event={"ID":"f2c55a01-06ca-407d-9f40-bcd70007c00a","Type":"ContainerDied","Data":"ef48ab90db94bd8cd69b60503cbbb97171e9f1e1cee3eb78dd92a0dff0c25f8a"} Oct 30 03:56:58.346786 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.336874 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef48ab90db94bd8cd69b60503cbbb97171e9f1e1cee3eb78dd92a0dff0c25f8a" Oct 30 03:56:58.378394 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.378358 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:56:58.536151 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.536052 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-metrics-certs\") pod \"f2c55a01-06ca-407d-9f40-bcd70007c00a\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " Oct 30 03:56:58.536475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.536453 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c55a01-06ca-407d-9f40-bcd70007c00a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c55a01-06ca-407d-9f40-bcd70007c00a-openshift-service-ca\") pod \"f2c55a01-06ca-407d-9f40-bcd70007c00a\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " Oct 30 03:56:58.537138 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.537009 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c55a01-06ca-407d-9f40-bcd70007c00a-openshift-service-ca" pod "f2c55a01-06ca-407d-9f40-bcd70007c00a" (UID: "f2c55a01-06ca-407d-9f40-bcd70007c00a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:56:58.537231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.537114 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c55a01-06ca-407d-9f40-bcd70007c00a-kube-api-access-n2dng\" (UniqueName: \"kubernetes.io/projected/f2c55a01-06ca-407d-9f40-bcd70007c00a-kube-api-access-n2dng\") pod \"f2c55a01-06ca-407d-9f40-bcd70007c00a\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " Oct 30 03:56:58.537305 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.537245 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-maas-api-tls\") pod \"f2c55a01-06ca-407d-9f40-bcd70007c00a\" (UID: \"f2c55a01-06ca-407d-9f40-bcd70007c00a\") " Oct 30 03:56:58.538581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.538553 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2c55a01-06ca-407d-9f40-bcd70007c00a-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:58.540624 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.540589 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c55a01-06ca-407d-9f40-bcd70007c00a-kube-api-access-n2dng" pod "f2c55a01-06ca-407d-9f40-bcd70007c00a" (UID: "f2c55a01-06ca-407d-9f40-bcd70007c00a"). InnerVolumeSpecName "kube-api-access-n2dng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:56:58.543448 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.542176 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-metrics-certs" pod "f2c55a01-06ca-407d-9f40-bcd70007c00a" (UID: "f2c55a01-06ca-407d-9f40-bcd70007c00a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:58.545304 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.545259 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-maas-api-tls" pod "f2c55a01-06ca-407d-9f40-bcd70007c00a" (UID: "f2c55a01-06ca-407d-9f40-bcd70007c00a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:56:58.639562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.639533 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2dng\" (UniqueName: \"kubernetes.io/projected/f2c55a01-06ca-407d-9f40-bcd70007c00a-kube-api-access-n2dng\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:58.639562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.639559 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:58.639562 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.639570 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2c55a01-06ca-407d-9f40-bcd70007c00a-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:56:58.696826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:58.696793 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:56:59.182699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:59.182652 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" podUID="5e77fc9f-bada-453e-8b68-b57512b8850b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:56:59.313375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:59.313342 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62" Oct 30 03:56:59.361326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:59.361295 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62"] Oct 30 03:56:59.371326 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:59.371303 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35fc87-68948d68b4-m9b62"] Oct 30 03:56:59.777334 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:56:59.777289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:00.109503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:00.109429 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c55a01-06ca-407d-9f40-bcd70007c00a" path="/var/lib/kubelet/pods/f2c55a01-06ca-407d-9f40-bcd70007c00a/volumes" Oct 30 03:57:00.215617 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:00.215586 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:01.106771 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.106713 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f"] Oct 30 03:57:01.107648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107623 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" Oct 30 03:57:01.107648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107646 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107664 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107670 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107680 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107686 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107704 4442 cpu_manager.go:414] "containerMap: removing container" podUID="174ed8a3-1a7f-45b0-816f-b5fd461b6a24" containerName="main" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107709 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="174ed8a3-1a7f-45b0-816f-b5fd461b6a24" containerName="main" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107727 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f2c55a01-06ca-407d-9f40-bcd70007c00a" containerName="maas-api" Oct 30 03:57:01.107832 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107732 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c55a01-06ca-407d-9f40-bcd70007c00a" containerName="maas-api" Oct 30 03:57:01.108279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107864 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="174ed8a3-1a7f-45b0-816f-b5fd461b6a24" containerName="main" Oct 30 03:57:01.108279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107876 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c55a01-06ca-407d-9f40-bcd70007c00a" containerName="maas-api" Oct 30 03:57:01.108279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107886 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfbb56a-d834-41fa-b559-09e8be196c7e" containerName="istio-proxy" Oct 30 03:57:01.108279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107894 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f78fc12-1193-44e0-ad47-d0db6a795b3a" containerName="istio-proxy" Oct 30 03:57:01.108279 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.107900 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9331b8fa-1b36-48c8-bb92-49ea25d8e66b" containerName="istio-proxy" Oct 30 03:57:01.130137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.130107 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f"] Oct 30 03:57:01.130328 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.130244 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.144442 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.144418 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0393ed\"/\"openshift-service-ca.crt\"" Oct 30 03:57:01.144541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.144526 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0393ed\"/\"kube-root-ca.crt\"" Oct 30 03:57:01.158481 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.158456 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0393ed\"/\"test-model-0393ed-kserve-self-signed-certs\"" Oct 30 03:57:01.158654 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.158528 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0393ed\"/\"default-dockercfg-98dxk\"" Oct 30 03:57:01.159596 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.159560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:01.289566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.289534 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-model-cache\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.289744 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.289609 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-dshm\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.289799 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.289736 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/51e4a7b5-0531-459c-90e0-cb31b9caecd7-kube-api-access-sbvmq\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.289897 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.289881 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tmp-dir\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.289970 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.289956 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-home\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.290013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.289988 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tls-certs\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.396845 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-home\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.396920 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tls-certs\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.397007 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-model-cache\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.397119 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-dshm\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.397241 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/51e4a7b5-0531-459c-90e0-cb31b9caecd7-kube-api-access-sbvmq\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.397391 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tmp-dir\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.398105 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tmp-dir\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.398349 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-home\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.402824 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.400959 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-model-cache\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.432516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.425500 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tls-certs\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.432516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.429883 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/51e4a7b5-0531-459c-90e0-cb31b9caecd7-kube-api-access-sbvmq\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.435675 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.434965 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-dshm\") pod \"test-model-0393ed-kserve-864d548475-qqf2f\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.451094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.451053 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:01.784924 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.784891 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f"] Oct 30 03:57:01.785055 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:57:01.784970 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51e4a7b5_0531_459c_90e0_cb31b9caecd7.slice/crio-0ee9e74b26db413a5e45ead7835429cad0c68068af9d88b83fdb7bf4dd153de2 WatchSource:0}: Error finding container 0ee9e74b26db413a5e45ead7835429cad0c68068af9d88b83fdb7bf4dd153de2: Status 404 returned error can't find the container with id 0ee9e74b26db413a5e45ead7835429cad0c68068af9d88b83fdb7bf4dd153de2 Oct 30 03:57:01.905203 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:01.905156 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:02.389575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:02.389531 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" event={"ID":"51e4a7b5-0531-459c-90e0-cb31b9caecd7","Type":"ContainerStarted","Data":"cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139"} Oct 30 03:57:02.390051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:02.389589 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" event={"ID":"51e4a7b5-0531-459c-90e0-cb31b9caecd7","Type":"ContainerStarted","Data":"0ee9e74b26db413a5e45ead7835429cad0c68068af9d88b83fdb7bf4dd153de2"} Oct 30 03:57:02.426705 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:02.426654 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" podStartSLOduration=1.426639611 podStartE2EDuration="1.426639611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:02.42513338 +0000 UTC m=+3246.427486576" watchObservedRunningTime="2026-10-30 03:57:02.426639611 +0000 UTC m=+3246.428992807" Oct 30 03:57:02.451632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:02.451581 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:03.396711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:03.396673 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:03.497619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:03.497586 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:03.597120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:03.597092 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:04.070979 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:04.070948 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:04.433207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:04.433104 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms"] Oct 30 03:57:04.433697 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:04.433323 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" containerID="cri-o://cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc" gracePeriod=30 Oct 30 03:57:04.433909 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:04.433883 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:05.145568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:05.145541 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:05.290065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:05.290009 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:05.384686 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:05.384651 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:05.484891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:05.484816 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:06.105193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:06.105165 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:06.796126 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:06.796097 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:08.384337 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:08.384303 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:08.459733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:08.459411 4442 patch_prober.go:29] interesting pod/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:57:08.459733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:08.459469 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:57:09.183216 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.183181 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" podUID="5e77fc9f-bada-453e-8b68-b57512b8850b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:57:09.734644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.734614 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:57:09.830077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830039 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-certs\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830288 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830096 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-envoy\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830373 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830308 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-socket\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830369 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-kube-api-access-t4jr6\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-kube-api-access-t4jr6\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830447 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830409 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-istio-token\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-istio-token\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830436 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-certs" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.830558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830460 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac681208-ea26-4098-ada6-0315accf7fcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac681208-ea26-4098-ada6-0315accf7fcf-istio-podinfo\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830558 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830542 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac681208-ea26-4098-ada6-0315accf7fcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac681208-ea26-4098-ada6-0315accf7fcf-istiod-ca-cert\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830646 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-credential-socket\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.830837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830677 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-socket" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.830837 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830691 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-data\") pod \"ac681208-ea26-4098-ada6-0315accf7fcf\" (UID: \"ac681208-ea26-4098-ada6-0315accf7fcf\") " Oct 30 03:57:09.831009 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.830970 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-data" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.831468 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.831438 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.831733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.831472 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.831733 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.831489 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.831912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.831866 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac681208-ea26-4098-ada6-0315accf7fcf-istiod-ca-cert" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:09.832137 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.832110 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-credential-socket" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.833600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.833574 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-kube-api-access-t4jr6" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "kube-api-access-t4jr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:09.833720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.833606 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac681208-ea26-4098-ada6-0315accf7fcf-istio-podinfo" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:57:09.833720 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.833683 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-envoy" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:57:09.833903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.833873 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-istio-token" pod "ac681208-ea26-4098-ada6-0315accf7fcf" (UID: "ac681208-ea26-4098-ada6-0315accf7fcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:09.906214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.906181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:09.933095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.933068 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jr6\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-kube-api-access-t4jr6\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.933095 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.933093 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac681208-ea26-4098-ada6-0315accf7fcf-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.933252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.933103 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac681208-ea26-4098-ada6-0315accf7fcf-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.933252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.933111 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac681208-ea26-4098-ada6-0315accf7fcf-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.933252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.933119 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:09.933252 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:09.933134 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac681208-ea26-4098-ada6-0315accf7fcf-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:10.528572 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.528540 4442 generic.go:356] "Generic (PLEG): container finished" podID="ac681208-ea26-4098-ada6-0315accf7fcf" containerID="cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc" exitCode=0 Oct 30 03:57:10.528747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.528592 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" Oct 30 03:57:10.528747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.528625 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" event={"ID":"ac681208-ea26-4098-ada6-0315accf7fcf","Type":"ContainerDied","Data":"cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc"} Oct 30 03:57:10.528747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.528687 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms" event={"ID":"ac681208-ea26-4098-ada6-0315accf7fcf","Type":"ContainerDied","Data":"962fdf180cc471fc09f290a3c293bf1c586b35650a3a712bf76bc5f680c7cc95"} Oct 30 03:57:10.528747 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.528716 4442 scope.go:122] "RemoveContainer" containerID="cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc" Oct 30 03:57:10.539528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.539510 4442 scope.go:122] "RemoveContainer" containerID="cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc" Oct 30 03:57:10.539814 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:57:10.539789 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc\": container with ID starting with cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc not found: ID does not exist" containerID="cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc" Oct 30 03:57:10.539939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.539818 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc"} err="failed to get container status \"cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc\": rpc error: code = NotFound desc = could not find container \"cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc\": container with ID starting with cd7667eb6d681d221dffa65030beae8a416c89d200e240ce8490781684753fbc not found: ID does not exist" Oct 30 03:57:10.588181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.588158 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms"] Oct 30 03:57:10.599295 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:10.599270 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-35fc87-openshift-default-6c54f7865c-kb8ms"] Oct 30 03:57:11.325418 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:11.325383 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:11.452566 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:11.452528 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:11.466232 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:11.466209 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:11.547486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:11.547456 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 03:57:12.115319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:12.115281 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" path="/var/lib/kubelet/pods/ac681208-ea26-4098-ada6-0315accf7fcf/volumes" Oct 30 03:57:12.808862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:12.808816 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:14.288579 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:14.288539 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:15.715181 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:15.715134 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:17.123543 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.123513 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:17.407839 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.407818 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:57:17.558070 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.558030 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e77fc9f-bada-453e-8b68-b57512b8850b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e77fc9f-bada-453e-8b68-b57512b8850b-plugins-config-volume\") pod \"5e77fc9f-bada-453e-8b68-b57512b8850b\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " Oct 30 03:57:17.558259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.558138 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e77fc9f-bada-453e-8b68-b57512b8850b-kube-api-access-8gdff\" (UniqueName: \"kubernetes.io/projected/5e77fc9f-bada-453e-8b68-b57512b8850b-kube-api-access-8gdff\") pod \"5e77fc9f-bada-453e-8b68-b57512b8850b\" (UID: \"5e77fc9f-bada-453e-8b68-b57512b8850b\") " Oct 30 03:57:17.558383 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.558361 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e77fc9f-bada-453e-8b68-b57512b8850b-plugins-config-volume" pod "5e77fc9f-bada-453e-8b68-b57512b8850b" (UID: "5e77fc9f-bada-453e-8b68-b57512b8850b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:57:17.558719 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.558698 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e77fc9f-bada-453e-8b68-b57512b8850b-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:17.560417 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.560395 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e77fc9f-bada-453e-8b68-b57512b8850b-kube-api-access-8gdff" pod "5e77fc9f-bada-453e-8b68-b57512b8850b" (UID: "5e77fc9f-bada-453e-8b68-b57512b8850b"). InnerVolumeSpecName "kube-api-access-8gdff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:57:17.585247 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.585222 4442 generic.go:356] "Generic (PLEG): container finished" podID="5e77fc9f-bada-453e-8b68-b57512b8850b" containerID="3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579" exitCode=0 Oct 30 03:57:17.585368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.585261 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" Oct 30 03:57:17.585368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.585302 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" event={"ID":"5e77fc9f-bada-453e-8b68-b57512b8850b","Type":"ContainerDied","Data":"3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579"} Oct 30 03:57:17.585368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.585351 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529" event={"ID":"5e77fc9f-bada-453e-8b68-b57512b8850b","Type":"ContainerDied","Data":"710abdf0fb3b04cf8ace9b1adc882d1f4b0c553d7f45fdcc41326381b43f69c7"} Oct 30 03:57:17.585512 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.585374 4442 scope.go:122] "RemoveContainer" containerID="3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579" Oct 30 03:57:17.595901 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.595880 4442 scope.go:122] "RemoveContainer" containerID="3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579" Oct 30 03:57:17.596172 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:57:17.596153 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579\": container with ID starting with 3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579 not found: ID does not exist" containerID="3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579" Oct 30 03:57:17.596254 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.596178 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579"} err="failed to get container status \"3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579\": rpc error: code = NotFound desc = could not find container \"3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579\": container with ID starting with 3c25815079d9d0ac7d15dbe7c1ba76947fdd53a6c173f8ad764c55cfa3130579 not found: ID does not exist" Oct 30 03:57:17.633529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.633477 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529"] Oct 30 03:57:17.648671 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.638894 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35fc87-784d6b744b-xp529"] Oct 30 03:57:17.659964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:17.659940 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gdff\" (UniqueName: \"kubernetes.io/projected/5e77fc9f-bada-453e-8b68-b57512b8850b-kube-api-access-8gdff\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:57:18.109358 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:18.109326 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e77fc9f-bada-453e-8b68-b57512b8850b" path="/var/lib/kubelet/pods/5e77fc9f-bada-453e-8b68-b57512b8850b/volumes" Oct 30 03:57:18.614469 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:18.614433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:20.284570 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:20.284536 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:21.926292 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:21.926213 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:22.457148 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.457110 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh"] Oct 30 03:57:22.457881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.457863 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" Oct 30 03:57:22.457881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.457881 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" Oct 30 03:57:22.458048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.457945 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5e77fc9f-bada-453e-8b68-b57512b8850b" containerName="payload-processing" Oct 30 03:57:22.458048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.457954 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e77fc9f-bada-453e-8b68-b57512b8850b" containerName="payload-processing" Oct 30 03:57:22.458162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.458125 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac681208-ea26-4098-ada6-0315accf7fcf" containerName="istio-proxy" Oct 30 03:57:22.458162 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.458149 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e77fc9f-bada-453e-8b68-b57512b8850b" containerName="payload-processing" Oct 30 03:57:22.469517 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.469492 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.484286 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.484255 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh"] Oct 30 03:57:22.487866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.487743 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b267f8\"/\"kube-root-ca.crt\"" Oct 30 03:57:22.487866 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.487835 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b267f8\"/\"openshift-service-ca.crt\"" Oct 30 03:57:22.494022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.493998 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b267f8\"/\"default-dockercfg-5ttg5\"" Oct 30 03:57:22.494129 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.494063 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b267f8\"/\"test-model-b267f8-kserve-self-signed-certs\"" Oct 30 03:57:22.496609 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.496584 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:22.560816 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.560780 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-home\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.560975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.560831 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-dshm\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.560975 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.560903 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tls-certs\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.561107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.560985 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tmp-dir\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.561107 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.561054 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-model-cache\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.561217 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.561145 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjl29\" (UniqueName: \"kubernetes.io/projected/2aeabff1-b68a-47b1-8010-ef639d3bf14c-kube-api-access-kjl29\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.662479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662443 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tmp-dir\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.662479 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662483 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-model-cache\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.662710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662528 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjl29\" (UniqueName: \"kubernetes.io/projected/2aeabff1-b68a-47b1-8010-ef639d3bf14c-kube-api-access-kjl29\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.662710 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662673 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-home\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.662881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662713 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-dshm\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.662881 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662791 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tls-certs\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.662973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662877 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tmp-dir\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.663015 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.662989 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-model-cache\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.663063 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.663051 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-home\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.665111 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.665090 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-dshm\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.665303 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.665286 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tls-certs\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.680218 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.680194 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjl29\" (UniqueName: \"kubernetes.io/projected/2aeabff1-b68a-47b1-8010-ef639d3bf14c-kube-api-access-kjl29\") pod \"test-model-b267f8-kserve-85fd48cc49-jzmvh\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:22.796782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:22.790086 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:23.137094 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:23.137065 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh"] Oct 30 03:57:23.137591 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:57:23.137567 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aeabff1_b68a_47b1_8010_ef639d3bf14c.slice/crio-eb66b5f426615fe3d65d28c891452e812088bb90a17d2a4aa87ffeede905473b WatchSource:0}: Error finding container eb66b5f426615fe3d65d28c891452e812088bb90a17d2a4aa87ffeede905473b: Status 404 returned error can't find the container with id eb66b5f426615fe3d65d28c891452e812088bb90a17d2a4aa87ffeede905473b Oct 30 03:57:23.609321 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:23.609289 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:23.734903 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:23.734860 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" event={"ID":"2aeabff1-b68a-47b1-8010-ef639d3bf14c","Type":"ContainerStarted","Data":"78bad4c9ca9e980a8a6ea23d3d8bad059db1701d5de101f54df17754620d73aa"} Oct 30 03:57:23.735146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:23.734917 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" event={"ID":"2aeabff1-b68a-47b1-8010-ef639d3bf14c","Type":"ContainerStarted","Data":"eb66b5f426615fe3d65d28c891452e812088bb90a17d2a4aa87ffeede905473b"} Oct 30 03:57:23.772140 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:23.772073 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" podStartSLOduration=1.7720041420000001 podStartE2EDuration="1.772004142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:23.770545903 +0000 UTC m=+3267.772899103" watchObservedRunningTime="2026-10-30 03:57:23.772004142 +0000 UTC m=+3267.774357340" Oct 30 03:57:23.790587 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:23.790566 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:24.862047 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:57:24.862009 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:57:24.968556 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:24.968523 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6"] Oct 30 03:57:24.973475 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:24.973452 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:24.980917 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:24.980882 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6"] Oct 30 03:57:24.985925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:24.985900 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:24.986826 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:24.986799 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx"] Oct 30 03:57:24.996013 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:24.995967 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6"] Oct 30 03:57:24.996128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:24.996068 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.020317 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.020290 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6"] Oct 30 03:57:25.041219 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.041191 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx"] Oct 30 03:57:25.111230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111196 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4nr\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-kube-api-access-rw4nr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111234 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111320 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111369 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111411 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111389 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111472 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111501 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lqd5\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-kube-api-access-2lqd5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111537 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111524 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111582 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111604 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111636 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.111704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111656 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111704 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111676 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.111820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111710 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111783 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111806 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.111912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111823 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111868 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.111977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111921 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111947 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.111977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111974 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.112090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.111997 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdlx\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-kube-api-access-vzdlx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.112090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.112074 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.112163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.112102 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.112163 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.112132 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.112227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.112162 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ebcfb-6be1-47de-95a2-4b8876205c34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.112227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.112194 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.213311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213277 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.213311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213317 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.213601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213339 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.213601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213415 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.213601 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213441 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lqd5\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-kube-api-access-2lqd5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.213912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213681 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.213912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213739 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.213912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213827 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.213912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213879 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.213912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213907 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.213975 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214009 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214015 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214051 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214091 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214122 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214156 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214179 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214235 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214264 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.214330 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214320 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214349 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdlx\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-kube-api-access-vzdlx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214400 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214434 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214508 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214530 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214602 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ebcfb-6be1-47de-95a2-4b8876205c34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214635 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214880 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214882 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4nr\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-kube-api-access-rw4nr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.215231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.214964 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.215817 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.215431 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.216193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.216166 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.216300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.216209 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ebcfb-6be1-47de-95a2-4b8876205c34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.216662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.216469 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.216662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.216640 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217117 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217188 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217378 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217407 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217384 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217602 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217616 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217649 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.218265 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.217879 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.218791 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.218733 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.218889 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.218874 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.219183 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.219163 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.231571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.231547 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.232185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.232165 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lqd5\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-kube-api-access-2lqd5\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-2kwq6\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.232513 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.232489 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.233179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.233154 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4nr\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-kube-api-access-rw4nr\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-28ksx\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.233793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.233771 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdlx\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-kube-api-access-vzdlx\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.234289 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.234265 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-55vj6\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.284168 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.284139 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:25.292187 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.292168 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:25.316173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.316152 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:25.321503 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.321481 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:25.639453 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.639425 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6"] Oct 30 03:57:25.640798 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:57:25.640736 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9cbdcf_886d_46e5_a7cd_7321e55e1a68.slice/crio-91c4329b113f48e5434d309c4795a315c7daf1a50dadb0adaa7337fa6feb2036 WatchSource:0}: Error finding container 91c4329b113f48e5434d309c4795a315c7daf1a50dadb0adaa7337fa6feb2036: Status 404 returned error can't find the container with id 91c4329b113f48e5434d309c4795a315c7daf1a50dadb0adaa7337fa6feb2036 Oct 30 03:57:25.643801 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.643738 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.643918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.643847 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.643918 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.643891 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.714575 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.714541 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx"] Oct 30 03:57:25.715812 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:57:25.715787 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7e4e5c_5372_4cea_aa2f_8d63671314de.slice/crio-bbd69ef800464efc8c2a74ec48bfef9f5251419c64e78e457dee2a37a730ebc0 WatchSource:0}: Error finding container bbd69ef800464efc8c2a74ec48bfef9f5251419c64e78e457dee2a37a730ebc0: Status 404 returned error can't find the container with id bbd69ef800464efc8c2a74ec48bfef9f5251419c64e78e457dee2a37a730ebc0 Oct 30 03:57:25.719581 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.719545 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.719679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.719631 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.719679 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.719673 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.733228 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:57:25.733200 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46ebcfb_6be1_47de_95a2_4b8876205c34.slice/crio-d62b5670ca6e04cfddeff79bcd613f8a0a3d4043c55465c2f328dcb88e982aee WatchSource:0}: Error finding container d62b5670ca6e04cfddeff79bcd613f8a0a3d4043c55465c2f328dcb88e982aee: Status 404 returned error can't find the container with id d62b5670ca6e04cfddeff79bcd613f8a0a3d4043c55465c2f328dcb88e982aee Oct 30 03:57:25.733381 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.733360 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6"] Oct 30 03:57:25.736820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.736784 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.736931 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.736902 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.737000 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.736949 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:57:25.754683 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.754651 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" event={"ID":"3f7e4e5c-5372-4cea-aa2f-8d63671314de","Type":"ContainerStarted","Data":"bbd69ef800464efc8c2a74ec48bfef9f5251419c64e78e457dee2a37a730ebc0"} Oct 30 03:57:25.756408 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.756381 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" event={"ID":"f46ebcfb-6be1-47de-95a2-4b8876205c34","Type":"ContainerStarted","Data":"d62b5670ca6e04cfddeff79bcd613f8a0a3d4043c55465c2f328dcb88e982aee"} Oct 30 03:57:25.759276 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.759225 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" event={"ID":"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68","Type":"ContainerStarted","Data":"583711099fc931555768c78b9656755b56fdbbdc1b45a088ee2beb35afa7ade9"} Oct 30 03:57:25.759359 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.759287 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" event={"ID":"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68","Type":"ContainerStarted","Data":"91c4329b113f48e5434d309c4795a315c7daf1a50dadb0adaa7337fa6feb2036"} Oct 30 03:57:25.803822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:25.803771 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podStartSLOduration=1.8037359080000002 podStartE2EDuration="1.803735908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:25.797319561 +0000 UTC m=+3269.799672760" watchObservedRunningTime="2026-10-30 03:57:25.803735908 +0000 UTC m=+3269.806089110" Oct 30 03:57:26.292706 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:26.292657 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:26.294281 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:26.294255 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:26.294415 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:26.294306 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:26.772656 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:26.772613 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" event={"ID":"f46ebcfb-6be1-47de-95a2-4b8876205c34","Type":"ContainerStarted","Data":"fa6e1030a84f5159fbdc2df76d4cefe13825ff4d7dd721799b6fa8f0e1708043"} Oct 30 03:57:26.774887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:26.774428 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" event={"ID":"3f7e4e5c-5372-4cea-aa2f-8d63671314de","Type":"ContainerStarted","Data":"3a47a859f7569008eb7767ad7be062775ce5d7776a9d5a3169c2c96bdcc8a996"} Oct 30 03:57:26.904242 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:26.904175 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podStartSLOduration=2.90415436 podStartE2EDuration="2.90415436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:26.856103673 +0000 UTC m=+3270.858456904" watchObservedRunningTime="2026-10-30 03:57:26.90415436 +0000 UTC m=+3270.906507549" Oct 30 03:57:26.923550 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:26.923479 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podStartSLOduration=2.923458631 podStartE2EDuration="2.923458631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:26.902722782 +0000 UTC m=+3270.905075982" watchObservedRunningTime="2026-10-30 03:57:26.923458631 +0000 UTC m=+3270.925811830" Oct 30 03:57:27.026856 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.026746 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:27.297836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.296192 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:27.297836 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.296255 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:27.330822 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.330784 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:27.332638 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.332614 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:27.332988 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.332959 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-28ksx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 03:57:27.333241 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.333216 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 03:57:27.333397 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.333151 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-55vj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 03:57:27.333521 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:27.333501 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 03:57:28.298782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:28.295296 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:28.298782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:28.295385 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:28.320229 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:28.320194 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-55vj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 03:57:28.320549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:28.320503 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 03:57:28.324179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:28.324138 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-28ksx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 03:57:28.324368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:28.324343 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 03:57:28.939782 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:28.936472 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:29.297777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:29.297026 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:29.297777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:29.297092 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:29.320780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:29.317314 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-55vj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 03:57:29.320780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:29.317379 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 03:57:29.332779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:29.322077 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-28ksx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 03:57:29.332779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:29.322127 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 03:57:30.300779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:30.298987 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:30.300779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:30.299044 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:30.320780 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:30.320692 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-55vj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 03:57:30.320969 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:30.320786 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 03:57:30.327779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:30.325071 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-28ksx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 03:57:30.327779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:30.325118 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 03:57:30.872741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:30.872704 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:31.295192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:31.295155 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:31.295506 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:31.295477 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:31.319259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:31.319137 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-55vj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 03:57:31.319259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:31.319197 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 03:57:31.323960 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:31.323928 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-28ksx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 03:57:31.324531 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:31.324502 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 03:57:32.264779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.264433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:32.293022 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.292989 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:32.293199 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.293043 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:32.317179 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.317144 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-55vj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 03:57:32.317342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.317211 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 03:57:32.322939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.322913 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-28ksx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 03:57:32.323052 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.322967 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 03:57:32.621600 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.621518 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:32.791090 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.791055 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:32.811414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.811383 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:32.900117 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:32.900026 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 03:57:33.293346 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:33.293306 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" start-of-body= Oct 30 03:57:33.293549 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:33.293375 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.220:15021/healthz/ready\": dial tcp 10.134.0.220:15021: connect: connection refused" Oct 30 03:57:33.316772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:33.316722 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-55vj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 03:57:33.316973 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:33.316805 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 03:57:33.322198 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:33.322165 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-28ksx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 03:57:33.322296 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:33.322226 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 03:57:34.329307 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:34.329040 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:34.714972 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:34.714825 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:34.815616 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:34.815581 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:35.101493 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.101448 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:35.103887 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.103855 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:35.105048 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.105011 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 03:57:35.198781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.197928 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:35.198781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.198781 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:35.199611 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.199587 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 03:57:35.291310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.287285 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:35.291310 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.288280 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:35.292109 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.292086 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 03:57:35.858118 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:35.858084 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:37.423900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:37.423866 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:38.807812 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:38.807776 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:39.189457 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:39.189347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:40.692311 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:40.692281 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:42.188221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:42.188192 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:43.709404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:43.709374 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:45.187990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:45.187960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:46.602036 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:46.602001 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:48.095100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:48.095067 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:49.600115 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:49.600073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:51.209030 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:51.209001 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:51.316987 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:51.316893 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:52.243045 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:52.243012 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:52.686156 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:52.686117 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:54.097929 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:54.097900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:54.203385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:54.203335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:54.862881 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:57:54.862845 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:57:55.686571 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:55.686537 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:56.635912 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:56.635886 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:57:56.640164 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:56.640137 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 03:57:57.186996 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:57.186957 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:57.419820 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:57.419789 4442 scope.go:122] "RemoveContainer" containerID="0c8572fdb925b4e382a529d5683462dc881af19675ba139256e3af46052a5bf5" Oct 30 03:57:57.429727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:57.429706 4442 scope.go:122] "RemoveContainer" containerID="fa287b11dc833270445ec787ade2d2a3a695513ca56574bd8167e82312b5f4ed" Oct 30 03:57:57.440573 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:57.440552 4442 scope.go:122] "RemoveContainer" containerID="8e5750cadc0210be1116d3c87f94d41a1bee971d0a0ce2caa4f45adf408210a8" Oct 30 03:57:58.731152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:58.731119 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:59.013385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:57:59.013350 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:58:00.564131 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:00.564097 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:02.151628 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:02.151591 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:03.749619 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:03.749587 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:05.208769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:05.208725 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:06.799690 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:06.799657 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:08.235385 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:08.235352 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:09.646789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:09.646746 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:11.213618 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.213582 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:11.439781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.439718 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks"] Oct 30 03:58:11.444932 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.444904 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.459741 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.459712 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b267f8-openshift-default-dockercfg-r8jvd\"" Oct 30 03:58:11.477648 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.477580 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks"] Oct 30 03:58:11.594505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.594461 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6896368e-e630-42b6-94a7-0f588f7c86bc-istiod-ca-cert\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.594703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.594592 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-credential-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.594703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.594637 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.594703 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.594679 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-envoy\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.594913 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.594849 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-podinfo\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.595044 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.595024 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-data\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.595088 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.595073 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-certs\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.595170 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.595156 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-token\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.595235 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.595219 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jz55\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-kube-api-access-2jz55\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697100 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-certs\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697143 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-token\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697316 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697183 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jz55\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-kube-api-access-2jz55\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697336 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6896368e-e630-42b6-94a7-0f588f7c86bc-istiod-ca-cert\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697373 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-credential-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697416 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-certs\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697417 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697534 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-envoy\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697587 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697632 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697618 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-podinfo\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697703 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-credential-socket\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.697827 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.697789 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-data\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.698100 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.698079 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-data\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.698177 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.698082 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6896368e-e630-42b6-94a7-0f588f7c86bc-istiod-ca-cert\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.699891 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.699862 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-envoy\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.700155 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.700135 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-podinfo\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.715259 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.715233 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-token\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.715352 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.715337 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jz55\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-kube-api-access-2jz55\") pod \"e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:11.760644 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:11.760621 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:12.080005 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.079976 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks"] Oct 30 03:58:12.080375 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:58:12.080354 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6896368e_e630_42b6_94a7_0f588f7c86bc.slice/crio-2ada24705f84e1f88a54d77f929e053f2867ae6ff75009bdf33c12a9b4d4b2f1 WatchSource:0}: Error finding container 2ada24705f84e1f88a54d77f929e053f2867ae6ff75009bdf33c12a9b4d4b2f1: Status 404 returned error can't find the container with id 2ada24705f84e1f88a54d77f929e053f2867ae6ff75009bdf33c12a9b4d4b2f1 Oct 30 03:58:12.083430 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.083398 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:58:12.083522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.083464 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:58:12.083522 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.083494 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 03:58:12.367591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.367492 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" event={"ID":"6896368e-e630-42b6-94a7-0f588f7c86bc","Type":"ContainerStarted","Data":"86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98"} Oct 30 03:58:12.367591 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.367562 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" event={"ID":"6896368e-e630-42b6-94a7-0f588f7c86bc","Type":"ContainerStarted","Data":"2ada24705f84e1f88a54d77f929e053f2867ae6ff75009bdf33c12a9b4d4b2f1"} Oct 30 03:58:12.410185 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.410117 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podStartSLOduration=1.4100971549999999 podStartE2EDuration="1.410097155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:12.405776426 +0000 UTC m=+3316.408129632" watchObservedRunningTime="2026-10-30 03:58:12.410097155 +0000 UTC m=+3316.412450356" Oct 30 03:58:12.697577 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.697492 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:12.789091 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:12.789051 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:13.763342 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:13.763308 4442 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 03:58:13.763900 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:13.763366 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 03:58:14.281772 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:14.281731 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:14.760990 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:14.760959 4442 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 03:58:14.761160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:14.761014 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 03:58:15.761825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:15.761793 4442 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 03:58:15.762223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:15.761847 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 03:58:15.930838 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:15.930806 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:16.761053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:16.761016 4442 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 03:58:16.761240 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:16.761070 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 03:58:17.513580 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:17.513547 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:17.761343 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:17.761310 4442 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 03:58:17.761514 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:17.761368 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 03:58:18.242375 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:18.242339 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:18.761056 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:18.761019 4442 patch_prober.go:29] interesting pod/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 03:58:18.761477 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:18.761076 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 03:58:19.058567 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:19.058486 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:19.766221 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:19.766195 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:19.766691 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:19.766658 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:19.767361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:19.767341 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 03:58:20.580769 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:20.580717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:22.158921 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:22.158890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:23.581999 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:23.581963 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:24.175516 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:24.175483 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:24.863904 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:58:24.863866 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:58:25.115674 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:25.115594 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:26.571014 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:26.570976 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:28.066193 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:28.066156 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:29.566128 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:29.566094 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:31.011789 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:31.011714 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:32.493172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:32.493141 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:33.904230 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:33.904194 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:35.374051 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:35.374020 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:36.876681 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:36.876629 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:37.134529 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:37.134450 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:37.843488 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:37.843453 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:38.463615 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:38.463577 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:39.859273 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:39.859235 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:41.455623 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:41.455572 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:43.243777 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:43.243220 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:45.080815 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:45.080778 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:46.510378 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:46.510342 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:46.597007 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:46.596964 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc"] Oct 30 03:58:46.597272 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:46.597245 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" containerID="cri-o://178175321abe2835ddbf5d9009ef9f654806a977df6b3f87ad7bc38f25f7cad3" gracePeriod=30 Oct 30 03:58:48.004699 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:48.004663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:49.545144 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:49.545104 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:51.098568 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:51.098524 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:51.399227 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:51.399151 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:51.598152 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:51.598120 4442 patch_prober.go:29] interesting pod/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 03:58:51.598335 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:51.598184 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:58:51.827564 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:51.827535 4442 generic.go:356] "Generic (PLEG): container finished" podID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerID="178175321abe2835ddbf5d9009ef9f654806a977df6b3f87ad7bc38f25f7cad3" exitCode=0 Oct 30 03:58:51.827779 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:51.827607 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" event={"ID":"f95105ce-6081-4317-8aaf-c38e36b7d770","Type":"ContainerDied","Data":"178175321abe2835ddbf5d9009ef9f654806a977df6b3f87ad7bc38f25f7cad3"} Oct 30 03:58:51.900287 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:51.900252 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:58:52.015192 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015162 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95105ce-6081-4317-8aaf-c38e36b7d770-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95105ce-6081-4317-8aaf-c38e36b7d770-istiod-ca-cert\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.015404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015255 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-certs\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.015404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015298 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-credential-socket\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.015404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015327 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-podinfo\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.015404 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015390 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-socket\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.015661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015518 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-kube-api-access-c2ch6\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-kube-api-access-c2ch6\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.015661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015581 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95105ce-6081-4317-8aaf-c38e36b7d770-istiod-ca-cert" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:58:52.015661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015598 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-credential-socket" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:52.015661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015606 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-certs" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:52.015661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015606 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-envoy\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.016010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015709 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-token\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-token\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.016010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015865 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-data\") pod \"f95105ce-6081-4317-8aaf-c38e36b7d770\" (UID: \"f95105ce-6081-4317-8aaf-c38e36b7d770\") " Oct 30 03:58:52.016010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.015877 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-socket" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:52.016925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.016840 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95105ce-6081-4317-8aaf-c38e36b7d770-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.016925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.016867 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.016925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.016880 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.016925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.016892 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.018851 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.018805 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-data" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:52.019119 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.019094 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-envoy" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:58:52.020709 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.020681 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-podinfo" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 03:58:52.021077 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.021044 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-token" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:52.021413 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.021391 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-kube-api-access-c2ch6" pod "f95105ce-6081-4317-8aaf-c38e36b7d770" (UID: "f95105ce-6081-4317-8aaf-c38e36b7d770"). InnerVolumeSpecName "kube-api-access-c2ch6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:58:52.118489 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.118463 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.118925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.118495 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ch6\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-kube-api-access-c2ch6\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.118925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.118509 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.118925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.118523 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.118925 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.118532 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95105ce-6081-4317-8aaf-c38e36b7d770-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:58:52.368271 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.368190 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:52.590351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.590320 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:52.836172 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.836144 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" Oct 30 03:58:52.836368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.836175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc" event={"ID":"f95105ce-6081-4317-8aaf-c38e36b7d770","Type":"ContainerDied","Data":"e84ce35271a5c9fa818b32e8a9f836844f9a53e0e10bfcb1352d59bcb43ea600"} Oct 30 03:58:52.836368 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.836231 4442 scope.go:122] "RemoveContainer" containerID="178175321abe2835ddbf5d9009ef9f654806a977df6b3f87ad7bc38f25f7cad3" Oct 30 03:58:52.898319 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.898288 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc"] Oct 30 03:58:52.909114 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:52.909090 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-855e14-openshift-default-5c9588dcd9-bp7bc"] Oct 30 03:58:53.991977 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:53.991946 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:54.121081 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:54.121049 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" path="/var/lib/kubelet/pods/f95105ce-6081-4317-8aaf-c38e36b7d770/volumes" Oct 30 03:58:54.154630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:54.154603 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:54.616500 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:54.616472 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:54.870679 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:58:54.870595 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:58:55.091382 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:55.091297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:55.509228 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:55.509194 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:56.974688 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:56.974659 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:58.395450 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:58.395416 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:59.802065 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:58:59.802033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:00.441329 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:00.441294 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c"] Oct 30 03:59:00.478127 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:00.478087 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf"] Oct 30 03:59:00.719199 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:00.719110 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:00.802189 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:00.802156 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:01.554961 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:01.554924 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:01.656215 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:01.656181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:02.730983 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:02.730953 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:02.829528 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:02.829487 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:03.141781 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:03.141735 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:04.694032 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:04.693997 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:05.441862 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.441818 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" podUID="abf331bc-81be-422e-bea5-34b16157cb52" containerName="payload-processing" containerID="cri-o://48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775" gracePeriod=25 Oct 30 03:59:05.479427 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.479370 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" podUID="9d989d15-7886-4cac-b022-13d584a23039" containerName="payload-pre-processing" containerID="cri-o://1e0d743525c5496e6f84b7a7a04f116962fa87ace29c92b98ae677543329de29" gracePeriod=25 Oct 30 03:59:05.716505 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.716440 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:59:05.845790 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.845738 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf331bc-81be-422e-bea5-34b16157cb52-kube-api-access-hqjj2\" (UniqueName: \"kubernetes.io/projected/abf331bc-81be-422e-bea5-34b16157cb52-kube-api-access-hqjj2\") pod \"abf331bc-81be-422e-bea5-34b16157cb52\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " Oct 30 03:59:05.846025 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.846012 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf331bc-81be-422e-bea5-34b16157cb52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf331bc-81be-422e-bea5-34b16157cb52-plugins-config-volume\") pod \"abf331bc-81be-422e-bea5-34b16157cb52\" (UID: \"abf331bc-81be-422e-bea5-34b16157cb52\") " Oct 30 03:59:05.846361 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.846341 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf331bc-81be-422e-bea5-34b16157cb52-plugins-config-volume" pod "abf331bc-81be-422e-bea5-34b16157cb52" (UID: "abf331bc-81be-422e-bea5-34b16157cb52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:59:05.846496 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.846482 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abf331bc-81be-422e-bea5-34b16157cb52-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:59:05.847962 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.847939 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf331bc-81be-422e-bea5-34b16157cb52-kube-api-access-hqjj2" pod "abf331bc-81be-422e-bea5-34b16157cb52" (UID: "abf331bc-81be-422e-bea5-34b16157cb52"). InnerVolumeSpecName "kube-api-access-hqjj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:59:05.947606 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.947573 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjj2\" (UniqueName: \"kubernetes.io/projected/abf331bc-81be-422e-bea5-34b16157cb52-kube-api-access-hqjj2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:59:05.951711 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.951683 4442 generic.go:356] "Generic (PLEG): container finished" podID="abf331bc-81be-422e-bea5-34b16157cb52" containerID="48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775" exitCode=0 Oct 30 03:59:05.951875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.951730 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" Oct 30 03:59:05.951875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.951778 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" event={"ID":"abf331bc-81be-422e-bea5-34b16157cb52","Type":"ContainerDied","Data":"48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775"} Oct 30 03:59:05.951875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.951837 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c" event={"ID":"abf331bc-81be-422e-bea5-34b16157cb52","Type":"ContainerDied","Data":"28f1b3e575be545aaef0e982e0572f81c629d3fe368c95481de9c8ef6378e448"} Oct 30 03:59:05.951875 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.951858 4442 scope.go:122] "RemoveContainer" containerID="48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775" Oct 30 03:59:05.963727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.963700 4442 scope.go:122] "RemoveContainer" containerID="48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775" Oct 30 03:59:05.964098 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:59:05.964079 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775\": container with ID starting with 48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775 not found: ID does not exist" containerID="48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775" Oct 30 03:59:05.964146 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:05.964107 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775"} err="failed to get container status \"48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775\": rpc error: code = NotFound desc = could not find container \"48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775\": container with ID starting with 48442f5af496bf15c65d33fa544c559ed30caf107eec34c884e82b7aca06f775 not found: ID does not exist" Oct 30 03:59:06.000201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:06.000177 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c"] Oct 30 03:59:06.003964 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:06.003929 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" podUID="9d989d15-7886-4cac-b022-13d584a23039" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:59:06.004267 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:06.004247 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3cfed7-79c7bbd5bd-kff8c"] Oct 30 03:59:06.110284 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:06.110256 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf331bc-81be-422e-bea5-34b16157cb52" path="/var/lib/kubelet/pods/abf331bc-81be-422e-bea5-34b16157cb52/volumes" Oct 30 03:59:06.243236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:06.243207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:07.407735 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:07.407698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:07.740075 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:07.739982 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:09.201871 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:09.201728 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:10.676902 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:10.676862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:12.063825 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:12.063789 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:12.162414 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:12.162378 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 03:59:13.564907 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:13.562917 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:14.095249 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:14.095199 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:15.683056 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:15.683020 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:16.004978 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:16.004946 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" podUID="9d989d15-7886-4cac-b022-13d584a23039" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 03:59:17.188208 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:17.188168 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:18.698106 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:18.698073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:19.520420 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:19.520388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:20.242793 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:20.242741 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:21.695236 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:21.695200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:23.273738 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:23.273704 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:24.684173 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:24.684138 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:24.862927 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:59:24.862893 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:59:25.121011 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:25.120980 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:25.569883 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:25.569843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:25.990402 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:25.990361 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:26.004495 ip-10-0-134-92 kubenswrapper[4442]: W1030 03:59:26.004196 4442 logging.go:55] [core] [Channel #10560 SubChannel #10561]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.196:50052", ServerName: "10.134.0.196:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.196:50052: connect: connection refused" Oct 30 03:59:26.117300 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.117261 4442 generic.go:356] "Generic (PLEG): container finished" podID="9d989d15-7886-4cac-b022-13d584a23039" containerID="1e0d743525c5496e6f84b7a7a04f116962fa87ace29c92b98ae677543329de29" exitCode=0 Oct 30 03:59:26.123222 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.123183 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" event={"ID":"9d989d15-7886-4cac-b022-13d584a23039","Type":"ContainerDied","Data":"1e0d743525c5496e6f84b7a7a04f116962fa87ace29c92b98ae677543329de29"} Oct 30 03:59:26.274237 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.274199 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:59:26.508876 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.508841 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d989d15-7886-4cac-b022-13d584a23039-kube-api-access-grwqr\" (UniqueName: \"kubernetes.io/projected/9d989d15-7886-4cac-b022-13d584a23039-kube-api-access-grwqr\") pod \"9d989d15-7886-4cac-b022-13d584a23039\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " Oct 30 03:59:26.509207 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.509181 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d989d15-7886-4cac-b022-13d584a23039-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d989d15-7886-4cac-b022-13d584a23039-plugins-config-volume\") pod \"9d989d15-7886-4cac-b022-13d584a23039\" (UID: \"9d989d15-7886-4cac-b022-13d584a23039\") " Oct 30 03:59:26.514223 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.514191 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d989d15-7886-4cac-b022-13d584a23039-plugins-config-volume" pod "9d989d15-7886-4cac-b022-13d584a23039" (UID: "9d989d15-7886-4cac-b022-13d584a23039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:59:26.555723 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.555648 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d989d15-7886-4cac-b022-13d584a23039-kube-api-access-grwqr" pod "9d989d15-7886-4cac-b022-13d584a23039" (UID: "9d989d15-7886-4cac-b022-13d584a23039"). InnerVolumeSpecName "kube-api-access-grwqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:59:26.611661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.611628 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grwqr\" (UniqueName: \"kubernetes.io/projected/9d989d15-7886-4cac-b022-13d584a23039-kube-api-access-grwqr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:59:26.611661 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.611656 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d989d15-7886-4cac-b022-13d584a23039-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 03:59:26.781239 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:26.781207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:27.004214 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:27.004174 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" podUID="9d989d15-7886-4cac-b022-13d584a23039" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.196:50052\" within 1s: context deadline exceeded" Oct 30 03:59:27.004630 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:27.004286 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:59:27.125784 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:27.125724 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" Oct 30 03:59:27.125984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:27.125783 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf" event={"ID":"9d989d15-7886-4cac-b022-13d584a23039","Type":"ContainerDied","Data":"99527bb53294cc4304a6a84a6a988d73e860af96948865c26f53096012e727af"} Oct 30 03:59:27.125984 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:27.125850 4442 scope.go:122] "RemoveContainer" containerID="1e0d743525c5496e6f84b7a7a04f116962fa87ace29c92b98ae677543329de29" Oct 30 03:59:27.195406 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:27.195375 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf"] Oct 30 03:59:27.199053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:27.199030 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3cfed7-65c8bc9bdc-l6kzf"] Oct 30 03:59:28.111823 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:28.111789 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d989d15-7886-4cac-b022-13d584a23039" path="/var/lib/kubelet/pods/9d989d15-7886-4cac-b022-13d584a23039/volumes" Oct 30 03:59:28.298560 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:28.298529 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:29.760952 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:29.760908 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:31.266180 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:31.266127 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:32.280244 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:32.280213 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:32.379715 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:32.379685 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:33.006465 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:33.006433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:33.141004 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:33.140972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:33.421635 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:33.421549 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:34.230010 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:34.229963 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:34.878662 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:34.878622 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:36.273958 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:36.273914 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:37.702160 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:37.702119 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:39.189490 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:39.189453 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:40.071727 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:40.071690 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:40.193120 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:40.193080 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:40.455201 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:40.455113 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:40.988053 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:40.988020 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:42.434541 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:42.434506 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:43.897351 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:43.897321 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:45.380266 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:45.380230 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:46.139939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:46.139906 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:46.755939 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:46.755907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:48.204892 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:48.204855 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:48.724508 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:48.724468 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:48.821274 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:48.821244 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:49.693486 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:49.693445 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:49.802345 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:49.802318 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:51.380610 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:51.380579 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:51.529256 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:51.529224 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:52.485966 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:52.485935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:52.806231 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:52.806198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:54.206407 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:54.206349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:54.321426 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:54.321389 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:54.862021 ip-10-0-134-92 kubenswrapper[4442]: E1030 03:59:54.861986 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 03:59:55.773631 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:55.773585 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:57.166773 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:57.166717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:58.562019 ip-10-0-134-92 kubenswrapper[4442]: I1030 03:59:58.561983 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:00.096513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.096479 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:00.221870 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.221822 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4"] Oct 30 04:00:00.223300 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223269 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" Oct 30 04:00:00.223453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223300 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" Oct 30 04:00:00.223453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223353 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9d989d15-7886-4cac-b022-13d584a23039" containerName="payload-pre-processing" Oct 30 04:00:00.223453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223362 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d989d15-7886-4cac-b022-13d584a23039" containerName="payload-pre-processing" Oct 30 04:00:00.223453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223451 4442 cpu_manager.go:414] "containerMap: removing container" podUID="abf331bc-81be-422e-bea5-34b16157cb52" containerName="payload-processing" Oct 30 04:00:00.223690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223459 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf331bc-81be-422e-bea5-34b16157cb52" containerName="payload-processing" Oct 30 04:00:00.223690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223661 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf331bc-81be-422e-bea5-34b16157cb52" containerName="payload-processing" Oct 30 04:00:00.223690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223683 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95105ce-6081-4317-8aaf-c38e36b7d770" containerName="istio-proxy" Oct 30 04:00:00.223958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.223696 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d989d15-7886-4cac-b022-13d584a23039" containerName="payload-pre-processing" Oct 30 04:00:00.232167 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.232129 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl"] Oct 30 04:00:00.234452 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.234427 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.249566 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.249533 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6"] Oct 30 04:00:00.249901 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.249686 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.261440 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.260630 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz"] Oct 30 04:00:00.261440 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.260777 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.270093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.270040 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nvf2f\"" Oct 30 04:00:00.287970 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.287944 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf"] Oct 30 04:00:00.288204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.288183 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.300015 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.299987 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4"] Oct 30 04:00:00.300177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.300035 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl"] Oct 30 04:00:00.300177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.300064 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6"] Oct 30 04:00:00.300177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.300074 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.300177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.300091 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq"] Oct 30 04:00:00.309248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.309220 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf"] Oct 30 04:00:00.309451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.309275 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq"] Oct 30 04:00:00.309451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.309303 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz"] Oct 30 04:00:00.309451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.309441 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.334617 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.334585 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49684\" (UniqueName: \"kubernetes.io/projected/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-kube-api-access-49684\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.334812 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.334649 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvc9\" (UniqueName: \"kubernetes.io/projected/5b967b29-70ea-4e31-8946-8c4b7ee77558-kube-api-access-vzvc9\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.334812 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.334728 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgvhm\" (UniqueName: \"kubernetes.io/projected/70550a44-ea69-4758-b024-8f86c4252f3b-kube-api-access-kgvhm\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.334966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.334832 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.334966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.334918 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febf3ae-a71e-4ef8-821e-76fd7b679d66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.335089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.334961 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hw2\" (UniqueName: \"kubernetes.io/projected/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-kube-api-access-z6hw2\") pod \"maas-api-key-cleanup-29888880-nbzbf\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.335089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.335009 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvmbv\" (UniqueName: \"kubernetes.io/projected/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-kube-api-access-cvmbv\") pod \"maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.335214 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.335114 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzvn\" (UniqueName: \"kubernetes.io/projected/4febf3ae-a71e-4ef8-821e-76fd7b679d66-kube-api-access-jqzvn\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.335635 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.335610 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b967b29-70ea-4e31-8946-8c4b7ee77558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.335823 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.335802 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888880-nbzbf\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.335910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.335868 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70550a44-ea69-4758-b024-8f86c4252f3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.335991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.335924 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.437497 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437461 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.437722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437518 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febf3ae-a71e-4ef8-821e-76fd7b679d66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.437722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437548 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hw2\" (UniqueName: \"kubernetes.io/projected/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-kube-api-access-z6hw2\") pod \"maas-api-key-cleanup-29888880-nbzbf\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.437722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437582 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvmbv\" (UniqueName: \"kubernetes.io/projected/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-kube-api-access-cvmbv\") pod \"maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.437722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437616 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzvn\" (UniqueName: \"kubernetes.io/projected/4febf3ae-a71e-4ef8-821e-76fd7b679d66-kube-api-access-jqzvn\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.438019 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437729 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b967b29-70ea-4e31-8946-8c4b7ee77558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.438019 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437835 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888880-nbzbf\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.438019 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437904 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70550a44-ea69-4758-b024-8f86c4252f3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.438019 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.437965 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.438265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.438051 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49684\" (UniqueName: \"kubernetes.io/projected/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-kube-api-access-49684\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.438265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.438114 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvc9\" (UniqueName: \"kubernetes.io/projected/5b967b29-70ea-4e31-8946-8c4b7ee77558-kube-api-access-vzvc9\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.438265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.438171 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgvhm\" (UniqueName: \"kubernetes.io/projected/70550a44-ea69-4758-b024-8f86c4252f3b-kube-api-access-kgvhm\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.438435 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.438291 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.439207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.439159 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.439464 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.439444 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b967b29-70ea-4e31-8946-8c4b7ee77558-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.439464 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.439456 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70550a44-ea69-4758-b024-8f86c4252f3b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.439677 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.439659 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888880-nbzbf\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.439994 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.439976 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febf3ae-a71e-4ef8-821e-76fd7b679d66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.457080 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.457042 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvc9\" (UniqueName: \"kubernetes.io/projected/5b967b29-70ea-4e31-8946-8c4b7ee77558-kube-api-access-vzvc9\") pod \"maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.458021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.457997 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hw2\" (UniqueName: \"kubernetes.io/projected/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-kube-api-access-z6hw2\") pod \"maas-api-key-cleanup-29888880-nbzbf\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.458132 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.458079 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49684\" (UniqueName: \"kubernetes.io/projected/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-kube-api-access-49684\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.458250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.458229 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvmbv\" (UniqueName: \"kubernetes.io/projected/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-kube-api-access-cvmbv\") pod \"maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.458374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.458316 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzvn\" (UniqueName: \"kubernetes.io/projected/4febf3ae-a71e-4ef8-821e-76fd7b679d66-kube-api-access-jqzvn\") pod \"maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.459137 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.459118 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgvhm\" (UniqueName: \"kubernetes.io/projected/70550a44-ea69-4758-b024-8f86c4252f3b-kube-api-access-kgvhm\") pod \"maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.580544 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.580511 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:00.588092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.588067 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:00.614020 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.611588 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:00.631158 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.631127 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:00.669826 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.669013 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:00.669826 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.669558 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:00.894506 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:00:00.894475 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70550a44_ea69_4758_b024_8f86c4252f3b.slice/crio-b2ed00b75c402bbee1cd0dd1220ea1a1f81a0b9646ca5342e120d074cc5f7945 WatchSource:0}: Error finding container b2ed00b75c402bbee1cd0dd1220ea1a1f81a0b9646ca5342e120d074cc5f7945: Status 404 returned error can't find the container with id b2ed00b75c402bbee1cd0dd1220ea1a1f81a0b9646ca5342e120d074cc5f7945 Oct 30 04:00:00.896379 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.896354 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4"] Oct 30 04:00:00.914957 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:00.914935 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl"] Oct 30 04:00:00.917436 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:00:00.917378 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b967b29_70ea_4e31_8946_8c4b7ee77558.slice/crio-e250292e6f3a60b0c054ba26aaaeb2a83bea2a7c7c6181fdcde96c8ffd6af167 WatchSource:0}: Error finding container e250292e6f3a60b0c054ba26aaaeb2a83bea2a7c7c6181fdcde96c8ffd6af167: Status 404 returned error can't find the container with id e250292e6f3a60b0c054ba26aaaeb2a83bea2a7c7c6181fdcde96c8ffd6af167 Oct 30 04:00:01.005286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.005259 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz"] Oct 30 04:00:01.036202 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.036174 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf"] Oct 30 04:00:01.043365 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.043342 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq"] Oct 30 04:00:01.086876 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:00:01.086839 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e6e45a_fb95_4bd6_9bf5_4362009ff96b.slice/crio-3778be630ede5d37b535635529fc82648ebc034c59bace90adab053aa9e735b8 WatchSource:0}: Error finding container 3778be630ede5d37b535635529fc82648ebc034c59bace90adab053aa9e735b8: Status 404 returned error can't find the container with id 3778be630ede5d37b535635529fc82648ebc034c59bace90adab053aa9e735b8 Oct 30 04:00:01.087723 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:00:01.087669 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7673a4d1_cd3f_4ba6_8a12_3a76eac7d16b.slice/crio-4178da1cb4ee8b8f309134ac3cf0f240bfb894214513b5f0e5351eb41c4b69ad WatchSource:0}: Error finding container 4178da1cb4ee8b8f309134ac3cf0f240bfb894214513b5f0e5351eb41c4b69ad: Status 404 returned error can't find the container with id 4178da1cb4ee8b8f309134ac3cf0f240bfb894214513b5f0e5351eb41c4b69ad Oct 30 04:00:01.089887 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:00:01.089847 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8b873e_8ca1_43c3_bcc6_aa9834bf39c2.slice/crio-8b0f82221e1433bb1429379b02d867c7d547a312f339503c209d4d909e712863 WatchSource:0}: Error finding container 8b0f82221e1433bb1429379b02d867c7d547a312f339503c209d4d909e712863: Status 404 returned error can't find the container with id 8b0f82221e1433bb1429379b02d867c7d547a312f339503c209d4d909e712863 Oct 30 04:00:01.149311 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:00:01.149279 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4febf3ae_a71e_4ef8_821e_76fd7b679d66.slice/crio-f806e77c94013093ffe72f13eff33ddf71853c13e6ab3b864a3aae9b2cacdb56 WatchSource:0}: Error finding container f806e77c94013093ffe72f13eff33ddf71853c13e6ab3b864a3aae9b2cacdb56: Status 404 returned error can't find the container with id f806e77c94013093ffe72f13eff33ddf71853c13e6ab3b864a3aae9b2cacdb56 Oct 30 04:00:01.150537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.150510 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6"] Oct 30 04:00:01.454459 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.454420 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" event={"ID":"4febf3ae-a71e-4ef8-821e-76fd7b679d66","Type":"ContainerStarted","Data":"f806e77c94013093ffe72f13eff33ddf71853c13e6ab3b864a3aae9b2cacdb56"} Oct 30 04:00:01.455885 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.455862 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" event={"ID":"5b967b29-70ea-4e31-8946-8c4b7ee77558","Type":"ContainerStarted","Data":"e250292e6f3a60b0c054ba26aaaeb2a83bea2a7c7c6181fdcde96c8ffd6af167"} Oct 30 04:00:01.457292 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.457268 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" event={"ID":"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b","Type":"ContainerStarted","Data":"4178da1cb4ee8b8f309134ac3cf0f240bfb894214513b5f0e5351eb41c4b69ad"} Oct 30 04:00:01.458936 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.458910 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" event={"ID":"51e6e45a-fb95-4bd6-9bf5-4362009ff96b","Type":"ContainerStarted","Data":"3778be630ede5d37b535635529fc82648ebc034c59bace90adab053aa9e735b8"} Oct 30 04:00:01.460675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.460645 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerStarted","Data":"6190d6f7717d430b2114922411991d1bb4893a26c402602215702579e3c1c7a1"} Oct 30 04:00:01.460823 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.460701 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerStarted","Data":"b2ed00b75c402bbee1cd0dd1220ea1a1f81a0b9646ca5342e120d074cc5f7945"} Oct 30 04:00:01.462145 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.462121 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" event={"ID":"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2","Type":"ContainerStarted","Data":"8b0f82221e1433bb1429379b02d867c7d547a312f339503c209d4d909e712863"} Oct 30 04:00:01.699474 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:01.699445 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:02.472453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.472420 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerID="ceca48954c77de3f58dd5271c760a21bcd3f9c99013687563cbea1e3b0b5fd79" exitCode=60 Oct 30 04:00:02.472934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.472500 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" event={"ID":"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2","Type":"ContainerDied","Data":"ceca48954c77de3f58dd5271c760a21bcd3f9c99013687563cbea1e3b0b5fd79"} Oct 30 04:00:02.472934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.472868 4442 scope.go:122] "RemoveContainer" containerID="ceca48954c77de3f58dd5271c760a21bcd3f9c99013687563cbea1e3b0b5fd79" Oct 30 04:00:02.474447 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.474425 4442 generic.go:356] "Generic (PLEG): container finished" podID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerID="5e53ea6cfc31c7ccc2e7a0f6f43c1bff6c3d9ce0a39041daf0da59e281fb51c6" exitCode=60 Oct 30 04:00:02.474594 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.474514 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" event={"ID":"4febf3ae-a71e-4ef8-821e-76fd7b679d66","Type":"ContainerDied","Data":"5e53ea6cfc31c7ccc2e7a0f6f43c1bff6c3d9ce0a39041daf0da59e281fb51c6"} Oct 30 04:00:02.474855 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.474831 4442 scope.go:122] "RemoveContainer" containerID="5e53ea6cfc31c7ccc2e7a0f6f43c1bff6c3d9ce0a39041daf0da59e281fb51c6" Oct 30 04:00:02.476616 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.476582 4442 generic.go:356] "Generic (PLEG): container finished" podID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerID="8e015d0c517d394a41ce6ad9abc43ce00840e2e3e2fd2c2c4fdf61a6957b0ef4" exitCode=60 Oct 30 04:00:02.476687 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.476674 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" event={"ID":"5b967b29-70ea-4e31-8946-8c4b7ee77558","Type":"ContainerDied","Data":"8e015d0c517d394a41ce6ad9abc43ce00840e2e3e2fd2c2c4fdf61a6957b0ef4"} Oct 30 04:00:02.476981 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.476965 4442 scope.go:122] "RemoveContainer" containerID="8e015d0c517d394a41ce6ad9abc43ce00840e2e3e2fd2c2c4fdf61a6957b0ef4" Oct 30 04:00:02.478903 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.478465 4442 generic.go:356] "Generic (PLEG): container finished" podID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerID="c2d18d8b452db31e5bfc5cfb0566ca3e2c7cb48205c4453aeaffec9ac84e72a5" exitCode=60 Oct 30 04:00:02.478903 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.478540 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" event={"ID":"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b","Type":"ContainerDied","Data":"c2d18d8b452db31e5bfc5cfb0566ca3e2c7cb48205c4453aeaffec9ac84e72a5"} Oct 30 04:00:02.478903 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.478714 4442 scope.go:122] "RemoveContainer" containerID="c2d18d8b452db31e5bfc5cfb0566ca3e2c7cb48205c4453aeaffec9ac84e72a5" Oct 30 04:00:02.481297 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.481278 4442 generic.go:356] "Generic (PLEG): container finished" podID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerID="513d32cb3eaf330d2bba16053d02d7824571c3098ecd0b6522e28de5b8ab107e" exitCode=60 Oct 30 04:00:02.481397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.481358 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" event={"ID":"51e6e45a-fb95-4bd6-9bf5-4362009ff96b","Type":"ContainerDied","Data":"513d32cb3eaf330d2bba16053d02d7824571c3098ecd0b6522e28de5b8ab107e"} Oct 30 04:00:02.481537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.481523 4442 scope.go:122] "RemoveContainer" containerID="513d32cb3eaf330d2bba16053d02d7824571c3098ecd0b6522e28de5b8ab107e" Oct 30 04:00:02.483184 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.483165 4442 generic.go:356] "Generic (PLEG): container finished" podID="70550a44-ea69-4758-b024-8f86c4252f3b" containerID="6190d6f7717d430b2114922411991d1bb4893a26c402602215702579e3c1c7a1" exitCode=60 Oct 30 04:00:02.483295 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.483212 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerDied","Data":"6190d6f7717d430b2114922411991d1bb4893a26c402602215702579e3c1c7a1"} Oct 30 04:00:02.483458 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.483441 4442 scope.go:122] "RemoveContainer" containerID="6190d6f7717d430b2114922411991d1bb4893a26c402602215702579e3c1c7a1" Oct 30 04:00:02.510869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:02.510810 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" podStartSLOduration=2.510790113 podStartE2EDuration="2.510790113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:01.496088669 +0000 UTC m=+3425.498441897" watchObservedRunningTime="2026-10-30 04:00:02.510790113 +0000 UTC m=+3426.513143324" Oct 30 04:00:03.274323 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.274291 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:03.496768 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.496710 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" event={"ID":"4febf3ae-a71e-4ef8-821e-76fd7b679d66","Type":"ContainerStarted","Data":"1affc9454028c4ca1c7e5fad89cdd6dbf180dc4026cc1b875355446578666a5c"} Oct 30 04:00:03.498822 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.498786 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" event={"ID":"5b967b29-70ea-4e31-8946-8c4b7ee77558","Type":"ContainerStarted","Data":"65cc020a101b1166d1a44ec9a153f6fe9d350543f749e451eab899e40602a076"} Oct 30 04:00:03.500597 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.500566 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" event={"ID":"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b","Type":"ContainerStarted","Data":"638ec8b31ea67cd36d741b05adb7e97acde8574a171df3f98c36a1b3b9fc7c1c"} Oct 30 04:00:03.502574 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.502549 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" event={"ID":"51e6e45a-fb95-4bd6-9bf5-4362009ff96b","Type":"ContainerStarted","Data":"7b965a91d55f6ebd530174a7ebe164b7cd2d943988fc695720f3a2c7c89f662f"} Oct 30 04:00:03.504504 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.504483 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerStarted","Data":"772313cb8374043829beb3a0cd5774be449dbd0616a0e4f9b267f5afa2c6086d"} Oct 30 04:00:03.506572 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.506549 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" event={"ID":"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2","Type":"ContainerStarted","Data":"262617fe604cd19935d5da1f14c6dcbf955599fe29facca2915012066edd44d2"} Oct 30 04:00:03.531461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.531389 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" podStartSLOduration=3.531380156 podStartE2EDuration="3.531380156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:03.53126057 +0000 UTC m=+3427.533613766" watchObservedRunningTime="2026-10-30 04:00:03.531380156 +0000 UTC m=+3427.533733353" Oct 30 04:00:03.566559 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.566519 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" podStartSLOduration=3.566506489 podStartE2EDuration="3.566506489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:03.565917554 +0000 UTC m=+3427.568270756" watchObservedRunningTime="2026-10-30 04:00:03.566506489 +0000 UTC m=+3427.568859685" Oct 30 04:00:03.606350 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.606305 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" podStartSLOduration=3.606291766 podStartE2EDuration="3.606291766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:03.601716149 +0000 UTC m=+3427.604069346" watchObservedRunningTime="2026-10-30 04:00:03.606291766 +0000 UTC m=+3427.608644963" Oct 30 04:00:03.636310 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.636262 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" podStartSLOduration=3.636248761 podStartE2EDuration="3.636248761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:03.63490388 +0000 UTC m=+3427.637257076" watchObservedRunningTime="2026-10-30 04:00:03.636248761 +0000 UTC m=+3427.638601957" Oct 30 04:00:03.670407 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:03.670368 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" podStartSLOduration=3.670355689 podStartE2EDuration="3.670355689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:03.669126584 +0000 UTC m=+3427.671479799" watchObservedRunningTime="2026-10-30 04:00:03.670355689 +0000 UTC m=+3427.672708886" Oct 30 04:00:04.517050 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.517017 4442 generic.go:356] "Generic (PLEG): container finished" podID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerID="65cc020a101b1166d1a44ec9a153f6fe9d350543f749e451eab899e40602a076" exitCode=60 Oct 30 04:00:04.517516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.517065 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" event={"ID":"5b967b29-70ea-4e31-8946-8c4b7ee77558","Type":"ContainerDied","Data":"65cc020a101b1166d1a44ec9a153f6fe9d350543f749e451eab899e40602a076"} Oct 30 04:00:04.517516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.517129 4442 scope.go:122] "RemoveContainer" containerID="8e015d0c517d394a41ce6ad9abc43ce00840e2e3e2fd2c2c4fdf61a6957b0ef4" Oct 30 04:00:04.517516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.517390 4442 scope.go:122] "RemoveContainer" containerID="65cc020a101b1166d1a44ec9a153f6fe9d350543f749e451eab899e40602a076" Oct 30 04:00:04.517740 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:04.517721 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl_odh-ai-gateway-infra(5b967b29-70ea-4e31-8946-8c4b7ee77558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" Oct 30 04:00:04.519333 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.519314 4442 generic.go:356] "Generic (PLEG): container finished" podID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerID="638ec8b31ea67cd36d741b05adb7e97acde8574a171df3f98c36a1b3b9fc7c1c" exitCode=60 Oct 30 04:00:04.519435 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.519386 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" event={"ID":"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b","Type":"ContainerDied","Data":"638ec8b31ea67cd36d741b05adb7e97acde8574a171df3f98c36a1b3b9fc7c1c"} Oct 30 04:00:04.519625 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.519606 4442 scope.go:122] "RemoveContainer" containerID="638ec8b31ea67cd36d741b05adb7e97acde8574a171df3f98c36a1b3b9fc7c1c" Oct 30 04:00:04.519948 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:04.519923 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888880-nbzbf_odh-ai-gateway-infra(7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" Oct 30 04:00:04.521810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.521786 4442 generic.go:356] "Generic (PLEG): container finished" podID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerID="7b965a91d55f6ebd530174a7ebe164b7cd2d943988fc695720f3a2c7c89f662f" exitCode=60 Oct 30 04:00:04.521916 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.521883 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" event={"ID":"51e6e45a-fb95-4bd6-9bf5-4362009ff96b","Type":"ContainerDied","Data":"7b965a91d55f6ebd530174a7ebe164b7cd2d943988fc695720f3a2c7c89f662f"} Oct 30 04:00:04.522284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.522263 4442 scope.go:122] "RemoveContainer" containerID="7b965a91d55f6ebd530174a7ebe164b7cd2d943988fc695720f3a2c7c89f662f" Oct 30 04:00:04.522588 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:04.522561 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz_odh-ai-gateway-infra(51e6e45a-fb95-4bd6-9bf5-4362009ff96b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" Oct 30 04:00:04.524149 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.524126 4442 generic.go:356] "Generic (PLEG): container finished" podID="70550a44-ea69-4758-b024-8f86c4252f3b" containerID="772313cb8374043829beb3a0cd5774be449dbd0616a0e4f9b267f5afa2c6086d" exitCode=60 Oct 30 04:00:04.524248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.524207 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerDied","Data":"772313cb8374043829beb3a0cd5774be449dbd0616a0e4f9b267f5afa2c6086d"} Oct 30 04:00:04.524444 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.524424 4442 scope.go:122] "RemoveContainer" containerID="772313cb8374043829beb3a0cd5774be449dbd0616a0e4f9b267f5afa2c6086d" Oct 30 04:00:04.524768 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:04.524724 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4_odh-ai-gateway-infra(70550a44-ea69-4758-b024-8f86c4252f3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" Oct 30 04:00:04.526827 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.526802 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerID="262617fe604cd19935d5da1f14c6dcbf955599fe29facca2915012066edd44d2" exitCode=60 Oct 30 04:00:04.527046 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.527026 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" event={"ID":"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2","Type":"ContainerDied","Data":"262617fe604cd19935d5da1f14c6dcbf955599fe29facca2915012066edd44d2"} Oct 30 04:00:04.527398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.527375 4442 scope.go:122] "RemoveContainer" containerID="262617fe604cd19935d5da1f14c6dcbf955599fe29facca2915012066edd44d2" Oct 30 04:00:04.527717 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:04.527634 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq_odh-ai-gateway-infra(9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" Oct 30 04:00:04.531553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.531528 4442 generic.go:356] "Generic (PLEG): container finished" podID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerID="1affc9454028c4ca1c7e5fad89cdd6dbf180dc4026cc1b875355446578666a5c" exitCode=60 Oct 30 04:00:04.531651 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.531574 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" event={"ID":"4febf3ae-a71e-4ef8-821e-76fd7b679d66","Type":"ContainerDied","Data":"1affc9454028c4ca1c7e5fad89cdd6dbf180dc4026cc1b875355446578666a5c"} Oct 30 04:00:04.531987 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.531968 4442 scope.go:122] "RemoveContainer" containerID="1affc9454028c4ca1c7e5fad89cdd6dbf180dc4026cc1b875355446578666a5c" Oct 30 04:00:04.532227 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:04.532197 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6_odh-ai-gateway-infra(4febf3ae-a71e-4ef8-821e-76fd7b679d66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" Oct 30 04:00:04.534550 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.534534 4442 scope.go:122] "RemoveContainer" containerID="c2d18d8b452db31e5bfc5cfb0566ca3e2c7cb48205c4453aeaffec9ac84e72a5" Oct 30 04:00:04.549645 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.549625 4442 scope.go:122] "RemoveContainer" containerID="513d32cb3eaf330d2bba16053d02d7824571c3098ecd0b6522e28de5b8ab107e" Oct 30 04:00:04.563515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.563497 4442 scope.go:122] "RemoveContainer" containerID="6190d6f7717d430b2114922411991d1bb4893a26c402602215702579e3c1c7a1" Oct 30 04:00:04.577084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.577065 4442 scope.go:122] "RemoveContainer" containerID="ceca48954c77de3f58dd5271c760a21bcd3f9c99013687563cbea1e3b0b5fd79" Oct 30 04:00:04.590461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.590442 4442 scope.go:122] "RemoveContainer" containerID="5e53ea6cfc31c7ccc2e7a0f6f43c1bff6c3d9ce0a39041daf0da59e281fb51c6" Oct 30 04:00:04.915438 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:04.915348 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:05.025779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:05.023331 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:05.543305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:05.543276 4442 scope.go:122] "RemoveContainer" containerID="1affc9454028c4ca1c7e5fad89cdd6dbf180dc4026cc1b875355446578666a5c" Oct 30 04:00:05.543790 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:05.543674 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6_odh-ai-gateway-infra(4febf3ae-a71e-4ef8-821e-76fd7b679d66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" Oct 30 04:00:05.545376 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:05.545352 4442 scope.go:122] "RemoveContainer" containerID="65cc020a101b1166d1a44ec9a153f6fe9d350543f749e451eab899e40602a076" Oct 30 04:00:05.545721 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:05.545696 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl_odh-ai-gateway-infra(5b967b29-70ea-4e31-8946-8c4b7ee77558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" Oct 30 04:00:05.547076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:05.547053 4442 scope.go:122] "RemoveContainer" containerID="638ec8b31ea67cd36d741b05adb7e97acde8574a171df3f98c36a1b3b9fc7c1c" Oct 30 04:00:05.547397 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:05.547373 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888880-nbzbf_odh-ai-gateway-infra(7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" Oct 30 04:00:05.548659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:05.548642 4442 scope.go:122] "RemoveContainer" containerID="7b965a91d55f6ebd530174a7ebe164b7cd2d943988fc695720f3a2c7c89f662f" Oct 30 04:00:05.548983 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:05.548964 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz_odh-ai-gateway-infra(51e6e45a-fb95-4bd6-9bf5-4362009ff96b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" Oct 30 04:00:05.550278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:05.550259 4442 scope.go:122] "RemoveContainer" containerID="772313cb8374043829beb3a0cd5774be449dbd0616a0e4f9b267f5afa2c6086d" Oct 30 04:00:05.550538 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:05.550516 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4_odh-ai-gateway-infra(70550a44-ea69-4758-b024-8f86c4252f3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" Oct 30 04:00:05.551969 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:05.551951 4442 scope.go:122] "RemoveContainer" containerID="262617fe604cd19935d5da1f14c6dcbf955599fe29facca2915012066edd44d2" Oct 30 04:00:05.552214 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:05.552193 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq_odh-ai-gateway-infra(9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" Oct 30 04:00:06.495579 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:06.495548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:07.040775 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:07.033207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:07.048090 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:07.048059 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:00:07.457380 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:07.457293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:07.694690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:07.694660 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:08.065524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:08.065488 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:09.684567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:09.684532 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:11.206072 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:11.206033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:11.811613 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:11.811579 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:12.705875 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:12.705836 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:13.185132 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:13.185092 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:13.294471 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:13.294412 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:13.819797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:13.816912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:14.368679 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:14.368641 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:15.100103 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:15.100074 4442 scope.go:122] "RemoveContainer" containerID="638ec8b31ea67cd36d741b05adb7e97acde8574a171df3f98c36a1b3b9fc7c1c" Oct 30 04:00:15.102634 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:15.100180 4442 scope.go:122] "RemoveContainer" containerID="7b965a91d55f6ebd530174a7ebe164b7cd2d943988fc695720f3a2c7c89f662f" Oct 30 04:00:15.102634 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:15.100504 4442 scope.go:122] "RemoveContainer" containerID="262617fe604cd19935d5da1f14c6dcbf955599fe29facca2915012066edd44d2" Oct 30 04:00:15.644323 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:15.644252 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" event={"ID":"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b","Type":"ContainerStarted","Data":"66101fef3a7798b1247852e6fe0df43b71921133f833475ad4173a4fd8195453"} Oct 30 04:00:15.646332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:15.646305 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" event={"ID":"51e6e45a-fb95-4bd6-9bf5-4362009ff96b","Type":"ContainerStarted","Data":"04fffb3821e8a02c4debc44b3b3a366bf07f934cce7991aaed223a8b0072e394"} Oct 30 04:00:15.648354 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:15.648265 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" event={"ID":"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2","Type":"ContainerStarted","Data":"45a5ed243a00569549e16ea2b0cf9254f9ba11b46bf1651a56d43c247eb361ae"} Oct 30 04:00:15.804123 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:15.804086 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:16.110991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.110965 4442 scope.go:122] "RemoveContainer" containerID="65cc020a101b1166d1a44ec9a153f6fe9d350543f749e451eab899e40602a076" Oct 30 04:00:16.658187 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.658148 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" event={"ID":"5b967b29-70ea-4e31-8946-8c4b7ee77558","Type":"ContainerStarted","Data":"b631154aa0ea6fd80e72f00342cbc5f6eaeef3004383b9b204c09df3df4be191"} Oct 30 04:00:16.659980 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.659957 4442 generic.go:356] "Generic (PLEG): container finished" podID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerID="66101fef3a7798b1247852e6fe0df43b71921133f833475ad4173a4fd8195453" exitCode=60 Oct 30 04:00:16.660130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.660025 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" event={"ID":"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b","Type":"ContainerDied","Data":"66101fef3a7798b1247852e6fe0df43b71921133f833475ad4173a4fd8195453"} Oct 30 04:00:16.660130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.660072 4442 scope.go:122] "RemoveContainer" containerID="638ec8b31ea67cd36d741b05adb7e97acde8574a171df3f98c36a1b3b9fc7c1c" Oct 30 04:00:16.660423 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.660406 4442 scope.go:122] "RemoveContainer" containerID="66101fef3a7798b1247852e6fe0df43b71921133f833475ad4173a4fd8195453" Oct 30 04:00:16.660798 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:16.660767 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888880-nbzbf_odh-ai-gateway-infra(7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" Oct 30 04:00:16.662088 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.662061 4442 generic.go:356] "Generic (PLEG): container finished" podID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerID="04fffb3821e8a02c4debc44b3b3a366bf07f934cce7991aaed223a8b0072e394" exitCode=60 Oct 30 04:00:16.662196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.662132 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" event={"ID":"51e6e45a-fb95-4bd6-9bf5-4362009ff96b","Type":"ContainerDied","Data":"04fffb3821e8a02c4debc44b3b3a366bf07f934cce7991aaed223a8b0072e394"} Oct 30 04:00:16.662490 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.662470 4442 scope.go:122] "RemoveContainer" containerID="04fffb3821e8a02c4debc44b3b3a366bf07f934cce7991aaed223a8b0072e394" Oct 30 04:00:16.662820 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:16.662801 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz_odh-ai-gateway-infra(51e6e45a-fb95-4bd6-9bf5-4362009ff96b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" Oct 30 04:00:16.664076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.664046 4442 generic.go:356] "Generic (PLEG): container finished" podID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerID="45a5ed243a00569549e16ea2b0cf9254f9ba11b46bf1651a56d43c247eb361ae" exitCode=60 Oct 30 04:00:16.664165 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.664113 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" event={"ID":"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2","Type":"ContainerDied","Data":"45a5ed243a00569549e16ea2b0cf9254f9ba11b46bf1651a56d43c247eb361ae"} Oct 30 04:00:16.664413 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.664390 4442 scope.go:122] "RemoveContainer" containerID="45a5ed243a00569549e16ea2b0cf9254f9ba11b46bf1651a56d43c247eb361ae" Oct 30 04:00:16.664672 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:16.664656 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq_odh-ai-gateway-infra(9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" Oct 30 04:00:16.673791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.673773 4442 scope.go:122] "RemoveContainer" containerID="7b965a91d55f6ebd530174a7ebe164b7cd2d943988fc695720f3a2c7c89f662f" Oct 30 04:00:16.687306 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.687291 4442 scope.go:122] "RemoveContainer" containerID="262617fe604cd19935d5da1f14c6dcbf955599fe29facca2915012066edd44d2" Oct 30 04:00:16.693115 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.693094 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf"] Oct 30 04:00:16.729972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.729937 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq"] Oct 30 04:00:16.761834 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:16.761741 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz"] Oct 30 04:00:17.054103 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.054012 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:17.100409 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.100379 4442 scope.go:122] "RemoveContainer" containerID="772313cb8374043829beb3a0cd5774be449dbd0616a0e4f9b267f5afa2c6086d" Oct 30 04:00:17.100572 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.100542 4442 scope.go:122] "RemoveContainer" containerID="1affc9454028c4ca1c7e5fad89cdd6dbf180dc4026cc1b875355446578666a5c" Oct 30 04:00:17.691308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.684700 4442 generic.go:356] "Generic (PLEG): container finished" podID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerID="b631154aa0ea6fd80e72f00342cbc5f6eaeef3004383b9b204c09df3df4be191" exitCode=60 Oct 30 04:00:17.691308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.684816 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" event={"ID":"5b967b29-70ea-4e31-8946-8c4b7ee77558","Type":"ContainerDied","Data":"b631154aa0ea6fd80e72f00342cbc5f6eaeef3004383b9b204c09df3df4be191"} Oct 30 04:00:17.691308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.684864 4442 scope.go:122] "RemoveContainer" containerID="65cc020a101b1166d1a44ec9a153f6fe9d350543f749e451eab899e40602a076" Oct 30 04:00:17.691308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.685321 4442 scope.go:122] "RemoveContainer" containerID="b631154aa0ea6fd80e72f00342cbc5f6eaeef3004383b9b204c09df3df4be191" Oct 30 04:00:17.691308 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:17.685742 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl_odh-ai-gateway-infra(5b967b29-70ea-4e31-8946-8c4b7ee77558)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" Oct 30 04:00:17.702005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.701970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerStarted","Data":"461256d9885e73fcd96487dbd8a51c1b62a4ca77230ef868d60fec8bbc7ca553"} Oct 30 04:00:17.718784 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.715047 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl"] Oct 30 04:00:17.923424 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.923357 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:17.928695 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:17.928668 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:18.066906 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.066875 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:18.075445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.075424 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:18.120251 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.120218 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-kube-api-access-49684\" (UniqueName: \"kubernetes.io/projected/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-kube-api-access-49684\") pod \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " Oct 30 04:00:18.120553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.120514 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-maas-api-service-ca\") pod \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\" (UID: \"51e6e45a-fb95-4bd6-9bf5-4362009ff96b\") " Oct 30 04:00:18.120942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.120918 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-maas-api-service-ca" pod "51e6e45a-fb95-4bd6-9bf5-4362009ff96b" (UID: "51e6e45a-fb95-4bd6-9bf5-4362009ff96b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:18.121133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.121104 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:18.122982 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.122956 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-kube-api-access-49684" pod "51e6e45a-fb95-4bd6-9bf5-4362009ff96b" (UID: "51e6e45a-fb95-4bd6-9bf5-4362009ff96b"). InnerVolumeSpecName "kube-api-access-49684". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:18.222608 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.222574 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-kube-api-access-z6hw2\" (UniqueName: \"kubernetes.io/projected/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-kube-api-access-z6hw2\") pod \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " Oct 30 04:00:18.222848 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.222660 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-maas-api-service-ca\") pod \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " Oct 30 04:00:18.222920 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.222851 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-kube-api-access-cvmbv\" (UniqueName: \"kubernetes.io/projected/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-kube-api-access-cvmbv\") pod \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\" (UID: \"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2\") " Oct 30 04:00:18.222971 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.222958 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-maas-api-service-ca\") pod \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\" (UID: \"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b\") " Oct 30 04:00:18.223089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.223060 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-maas-api-service-ca" pod "9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" (UID: "9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:18.223248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.223230 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-maas-api-service-ca" pod "7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" (UID: "7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:18.223904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.223882 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:18.223987 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.223910 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49684\" (UniqueName: \"kubernetes.io/projected/51e6e45a-fb95-4bd6-9bf5-4362009ff96b-kube-api-access-49684\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:18.223987 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.223925 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:18.225066 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.225045 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-kube-api-access-cvmbv" pod "9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" (UID: "9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2"). InnerVolumeSpecName "kube-api-access-cvmbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:18.225138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.225082 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-kube-api-access-z6hw2" pod "7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" (UID: "7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b"). InnerVolumeSpecName "kube-api-access-z6hw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:18.325638 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.325607 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvmbv\" (UniqueName: \"kubernetes.io/projected/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2-kube-api-access-cvmbv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:18.325638 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.325634 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hw2\" (UniqueName: \"kubernetes.io/projected/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b-kube-api-access-z6hw2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:18.712371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.712332 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" event={"ID":"7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b","Type":"ContainerDied","Data":"4178da1cb4ee8b8f309134ac3cf0f240bfb894214513b5f0e5351eb41c4b69ad"} Oct 30 04:00:18.712850 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.712378 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf" Oct 30 04:00:18.712850 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.712433 4442 scope.go:122] "RemoveContainer" containerID="66101fef3a7798b1247852e6fe0df43b71921133f833475ad4173a4fd8195453" Oct 30 04:00:18.714020 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.713976 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" Oct 30 04:00:18.714318 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.713974 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz" event={"ID":"51e6e45a-fb95-4bd6-9bf5-4362009ff96b","Type":"ContainerDied","Data":"3778be630ede5d37b535635529fc82648ebc034c59bace90adab053aa9e735b8"} Oct 30 04:00:18.716527 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.716499 4442 generic.go:356] "Generic (PLEG): container finished" podID="70550a44-ea69-4758-b024-8f86c4252f3b" containerID="461256d9885e73fcd96487dbd8a51c1b62a4ca77230ef868d60fec8bbc7ca553" exitCode=60 Oct 30 04:00:18.716669 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.716602 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerDied","Data":"461256d9885e73fcd96487dbd8a51c1b62a4ca77230ef868d60fec8bbc7ca553"} Oct 30 04:00:18.717066 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.717046 4442 scope.go:122] "RemoveContainer" containerID="461256d9885e73fcd96487dbd8a51c1b62a4ca77230ef868d60fec8bbc7ca553" Oct 30 04:00:18.717386 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:18.717362 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4_odh-ai-gateway-infra(70550a44-ea69-4758-b024-8f86c4252f3b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" Oct 30 04:00:18.718399 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.718377 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" Oct 30 04:00:18.718576 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.718493 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq" event={"ID":"9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2","Type":"ContainerDied","Data":"8b0f82221e1433bb1429379b02d867c7d547a312f339503c209d4d909e712863"} Oct 30 04:00:18.721989 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.721963 4442 generic.go:356] "Generic (PLEG): container finished" podID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerID="1a083419545a60e1a18040973fb4ff2495e09343c68c553985f9f5eea87875f3" exitCode=60 Oct 30 04:00:18.722093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.721987 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" event={"ID":"4febf3ae-a71e-4ef8-821e-76fd7b679d66","Type":"ContainerDied","Data":"1a083419545a60e1a18040973fb4ff2495e09343c68c553985f9f5eea87875f3"} Oct 30 04:00:18.722430 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.722411 4442 scope.go:122] "RemoveContainer" containerID="1a083419545a60e1a18040973fb4ff2495e09343c68c553985f9f5eea87875f3" Oct 30 04:00:18.722832 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:18.722805 4442 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6_odh-ai-gateway-infra(4febf3ae-a71e-4ef8-821e-76fd7b679d66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" Oct 30 04:00:18.727873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.727856 4442 scope.go:122] "RemoveContainer" containerID="04fffb3821e8a02c4debc44b3b3a366bf07f934cce7991aaed223a8b0072e394" Oct 30 04:00:18.739060 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.739041 4442 scope.go:122] "RemoveContainer" containerID="772313cb8374043829beb3a0cd5774be449dbd0616a0e4f9b267f5afa2c6086d" Oct 30 04:00:18.758868 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.758838 4442 scope.go:122] "RemoveContainer" containerID="45a5ed243a00569549e16ea2b0cf9254f9ba11b46bf1651a56d43c247eb361ae" Oct 30 04:00:18.768625 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.768604 4442 scope.go:122] "RemoveContainer" containerID="1affc9454028c4ca1c7e5fad89cdd6dbf180dc4026cc1b875355446578666a5c" Oct 30 04:00:18.785111 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.784746 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq"] Oct 30 04:00:18.789951 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.789925 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4206d9-29888880-jsjvq"] Oct 30 04:00:18.818630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.818595 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz"] Oct 30 04:00:18.823872 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.823845 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-852107-29888880-t24bz"] Oct 30 04:00:18.887864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.887834 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4"] Oct 30 04:00:18.894200 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.894171 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:18.920798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.920770 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf"] Oct 30 04:00:18.931985 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:18.931957 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888880-nbzbf"] Oct 30 04:00:19.036926 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.036898 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b967b29-70ea-4e31-8946-8c4b7ee77558-kube-api-access-vzvc9\" (UniqueName: \"kubernetes.io/projected/5b967b29-70ea-4e31-8946-8c4b7ee77558-kube-api-access-vzvc9\") pod \"5b967b29-70ea-4e31-8946-8c4b7ee77558\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " Oct 30 04:00:19.037109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.037036 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b967b29-70ea-4e31-8946-8c4b7ee77558-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b967b29-70ea-4e31-8946-8c4b7ee77558-maas-api-service-ca\") pod \"5b967b29-70ea-4e31-8946-8c4b7ee77558\" (UID: \"5b967b29-70ea-4e31-8946-8c4b7ee77558\") " Oct 30 04:00:19.037542 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.037514 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b967b29-70ea-4e31-8946-8c4b7ee77558-maas-api-service-ca" pod "5b967b29-70ea-4e31-8946-8c4b7ee77558" (UID: "5b967b29-70ea-4e31-8946-8c4b7ee77558"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:19.039492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.039465 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b967b29-70ea-4e31-8946-8c4b7ee77558-kube-api-access-vzvc9" pod "5b967b29-70ea-4e31-8946-8c4b7ee77558" (UID: "5b967b29-70ea-4e31-8946-8c4b7ee77558"). InnerVolumeSpecName "kube-api-access-vzvc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:19.139378 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.139337 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzvc9\" (UniqueName: \"kubernetes.io/projected/5b967b29-70ea-4e31-8946-8c4b7ee77558-kube-api-access-vzvc9\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:19.139378 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.139370 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b967b29-70ea-4e31-8946-8c4b7ee77558-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:19.583786 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.583736 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:19.732176 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.732149 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" Oct 30 04:00:19.732690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.732227 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl" event={"ID":"5b967b29-70ea-4e31-8946-8c4b7ee77558","Type":"ContainerDied","Data":"e250292e6f3a60b0c054ba26aaaeb2a83bea2a7c7c6181fdcde96c8ffd6af167"} Oct 30 04:00:19.732690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.732294 4442 scope.go:122] "RemoveContainer" containerID="b631154aa0ea6fd80e72f00342cbc5f6eaeef3004383b9b204c09df3df4be191" Oct 30 04:00:19.784235 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.784198 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl"] Oct 30 04:00:19.795533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.795507 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-93b821-29888880-4cvdl"] Oct 30 04:00:19.897873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.897834 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6"] Oct 30 04:00:19.909825 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:19.909796 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:20.065075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.065045 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70550a44-ea69-4758-b024-8f86c4252f3b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70550a44-ea69-4758-b024-8f86c4252f3b-maas-api-service-ca\") pod \"70550a44-ea69-4758-b024-8f86c4252f3b\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " Oct 30 04:00:20.065232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.065185 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70550a44-ea69-4758-b024-8f86c4252f3b-kube-api-access-kgvhm\" (UniqueName: \"kubernetes.io/projected/70550a44-ea69-4758-b024-8f86c4252f3b-kube-api-access-kgvhm\") pod \"70550a44-ea69-4758-b024-8f86c4252f3b\" (UID: \"70550a44-ea69-4758-b024-8f86c4252f3b\") " Oct 30 04:00:20.065508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.065480 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70550a44-ea69-4758-b024-8f86c4252f3b-maas-api-service-ca" pod "70550a44-ea69-4758-b024-8f86c4252f3b" (UID: "70550a44-ea69-4758-b024-8f86c4252f3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:20.065894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.065875 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70550a44-ea69-4758-b024-8f86c4252f3b-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:20.067586 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.067564 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70550a44-ea69-4758-b024-8f86c4252f3b-kube-api-access-kgvhm" pod "70550a44-ea69-4758-b024-8f86c4252f3b" (UID: "70550a44-ea69-4758-b024-8f86c4252f3b"). InnerVolumeSpecName "kube-api-access-kgvhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:20.093265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.093197 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:20.109932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.109904 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" path="/var/lib/kubelet/pods/51e6e45a-fb95-4bd6-9bf5-4362009ff96b/volumes" Oct 30 04:00:20.110455 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.110438 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" path="/var/lib/kubelet/pods/5b967b29-70ea-4e31-8946-8c4b7ee77558/volumes" Oct 30 04:00:20.110858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.110845 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" path="/var/lib/kubelet/pods/7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b/volumes" Oct 30 04:00:20.111542 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.111528 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" path="/var/lib/kubelet/pods/9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2/volumes" Oct 30 04:00:20.167267 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.167183 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4febf3ae-a71e-4ef8-821e-76fd7b679d66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febf3ae-a71e-4ef8-821e-76fd7b679d66-maas-api-service-ca\") pod \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " Oct 30 04:00:20.167568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.167545 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4febf3ae-a71e-4ef8-821e-76fd7b679d66-maas-api-service-ca" pod "4febf3ae-a71e-4ef8-821e-76fd7b679d66" (UID: "4febf3ae-a71e-4ef8-821e-76fd7b679d66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:20.167687 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.167579 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4febf3ae-a71e-4ef8-821e-76fd7b679d66-kube-api-access-jqzvn\" (UniqueName: \"kubernetes.io/projected/4febf3ae-a71e-4ef8-821e-76fd7b679d66-kube-api-access-jqzvn\") pod \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\" (UID: \"4febf3ae-a71e-4ef8-821e-76fd7b679d66\") " Oct 30 04:00:20.168097 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.168079 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgvhm\" (UniqueName: \"kubernetes.io/projected/70550a44-ea69-4758-b024-8f86c4252f3b-kube-api-access-kgvhm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:20.168174 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.168102 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4febf3ae-a71e-4ef8-821e-76fd7b679d66-maas-api-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:20.169916 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.169896 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4febf3ae-a71e-4ef8-821e-76fd7b679d66-kube-api-access-jqzvn" pod "4febf3ae-a71e-4ef8-821e-76fd7b679d66" (UID: "4febf3ae-a71e-4ef8-821e-76fd7b679d66"). InnerVolumeSpecName "kube-api-access-jqzvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:20.269249 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.269215 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqzvn\" (UniqueName: \"kubernetes.io/projected/4febf3ae-a71e-4ef8-821e-76fd7b679d66-kube-api-access-jqzvn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:20.714951 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.714915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:20.753815 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.753790 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" Oct 30 04:00:20.754273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.753824 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4" event={"ID":"70550a44-ea69-4758-b024-8f86c4252f3b","Type":"ContainerDied","Data":"b2ed00b75c402bbee1cd0dd1220ea1a1f81a0b9646ca5342e120d074cc5f7945"} Oct 30 04:00:20.754273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.753898 4442 scope.go:122] "RemoveContainer" containerID="461256d9885e73fcd96487dbd8a51c1b62a4ca77230ef868d60fec8bbc7ca553" Oct 30 04:00:20.756841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.756024 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" Oct 30 04:00:20.756841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.756042 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6" event={"ID":"4febf3ae-a71e-4ef8-821e-76fd7b679d66","Type":"ContainerDied","Data":"f806e77c94013093ffe72f13eff33ddf71853c13e6ab3b864a3aae9b2cacdb56"} Oct 30 04:00:20.768810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.768791 4442 scope.go:122] "RemoveContainer" containerID="1a083419545a60e1a18040973fb4ff2495e09343c68c553985f9f5eea87875f3" Oct 30 04:00:20.801007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.800985 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4"] Oct 30 04:00:20.801546 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.801523 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:20.811742 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.811717 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-689bcf-29888880-2zjf4"] Oct 30 04:00:20.851144 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.851099 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6"] Oct 30 04:00:20.856286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:20.856265 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9d8692-29888880-zdzd6"] Oct 30 04:00:21.132730 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:21.132698 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:21.544802 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:21.542374 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:21.987350 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:21.987255 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:22.097806 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:22.097368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:22.118917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:22.118881 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" path="/var/lib/kubelet/pods/4febf3ae-a71e-4ef8-821e-76fd7b679d66/volumes" Oct 30 04:00:22.119350 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:22.119332 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" path="/var/lib/kubelet/pods/70550a44-ea69-4758-b024-8f86c4252f3b/volumes" Oct 30 04:00:22.753731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:22.753674 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:22.888474 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:22.888404 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:23.140854 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:23.140724 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25"] Oct 30 04:00:23.141644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:23.141378 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" containerID="cri-o://f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1" gracePeriod=30 Oct 30 04:00:23.155265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:23.155238 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2"] Oct 30 04:00:23.155720 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:23.155687 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" containerID="cri-o://91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b" gracePeriod=30 Oct 30 04:00:23.156087 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:23.156067 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:23.161477 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:23.161449 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6"] Oct 30 04:00:23.161689 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:23.161667 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" containerID="cri-o://a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4" gracePeriod=30 Oct 30 04:00:24.082204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:24.082169 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:24.176338 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:24.176307 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:24.276626 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:24.276598 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:24.700495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:24.700440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:24.862733 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:24.862694 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:00:25.198211 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:25.198174 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:25.296917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:25.296886 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:25.398584 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:25.398552 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:26.397646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:26.396767 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:27.815073 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:27.815037 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:28.244810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.244778 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72"] Oct 30 04:00:28.244994 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.244971 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" containerID="cri-o://a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8" gracePeriod=30 Oct 30 04:00:28.479816 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.479028 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 04:00:28.495367 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.495342 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 04:00:28.512848 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.512821 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-istio-token\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-istio-token\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.512884 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-token\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-token\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.512917 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-credential-socket\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.512994 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-envoy\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513032 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-kube-api-access-8cw87\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-kube-api-access-8cw87\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513091 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-data\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513127 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-credential-socket\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513163 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf49285b-876e-4a6d-8c3e-c8602687306d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49285b-876e-4a6d-8c3e-c8602687306d-istiod-ca-cert\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513207 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-envoy\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513246 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89464194-c498-4655-8868-a930de543542-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89464194-c498-4655-8868-a930de543542-istio-podinfo\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513285 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-data\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513337 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-socket\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513380 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-socket\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513455 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-kube-api-access-ms7mv\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-kube-api-access-ms7mv\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513543 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89464194-c498-4655-8868-a930de543542-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89464194-c498-4655-8868-a930de543542-istiod-ca-cert\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513606 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-certs\") pod \"89464194-c498-4655-8868-a930de543542\" (UID: \"89464194-c498-4655-8868-a930de543542\") " Oct 30 04:00:28.513922 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513710 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-certs\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.513922 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513787 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-podinfo\") pod \"bf49285b-876e-4a6d-8c3e-c8602687306d\" (UID: \"bf49285b-876e-4a6d-8c3e-c8602687306d\") " Oct 30 04:00:28.514036 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.513946 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-credential-socket" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.514108 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.514090 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-data" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.514389 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.514373 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.514468 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.514392 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.514660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.514605 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-data" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.514972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.514953 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf49285b-876e-4a6d-8c3e-c8602687306d-istiod-ca-cert" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:28.515227 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.515210 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-credential-socket" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.520149 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.519039 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-socket" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.520149 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.519137 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89464194-c498-4655-8868-a930de543542-istiod-ca-cert" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:28.520149 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.519235 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-certs" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.520149 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.519386 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-certs" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.520149 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.519478 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-socket" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.528454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.528415 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-envoy" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.528563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.528483 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-envoy" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.528940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.528900 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-podinfo" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:00:28.529046 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.529029 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-istio-token" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.529189 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.529140 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-kube-api-access-8cw87" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "kube-api-access-8cw87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.529455 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.529428 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89464194-c498-4655-8868-a930de543542-istio-podinfo" pod "89464194-c498-4655-8868-a930de543542" (UID: "89464194-c498-4655-8868-a930de543542"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:00:28.529553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.529481 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-kube-api-access-ms7mv" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "kube-api-access-ms7mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.530039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.530019 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-token" pod "bf49285b-876e-4a6d-8c3e-c8602687306d" (UID: "bf49285b-876e-4a6d-8c3e-c8602687306d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.563422 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.563391 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.615518 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e036dea4-c6a2-4956-a75c-8f5f0904d287-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e036dea4-c6a2-4956-a75c-8f5f0904d287-istiod-ca-cert\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.615642 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-podinfo\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.615732 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-certs\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.615899 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-token\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-token\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.615999 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-kube-api-access-swdbr\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-kube-api-access-swdbr\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616028 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-data\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616090 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-socket\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616092 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-certs" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616191 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-envoy\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.616312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616250 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-credential-socket\") pod \"e036dea4-c6a2-4956-a75c-8f5f0904d287\" (UID: \"e036dea4-c6a2-4956-a75c-8f5f0904d287\") " Oct 30 04:00:28.617029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616453 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-data" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.617029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616623 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e036dea4-c6a2-4956-a75c-8f5f0904d287-istiod-ca-cert" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:28.617029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.616767 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-credential-socket" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617067 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617091 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617105 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617118 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617130 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cw87\" (UniqueName: \"kubernetes.io/projected/89464194-c498-4655-8868-a930de543542-kube-api-access-8cw87\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617143 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617158 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617170 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf49285b-876e-4a6d-8c3e-c8602687306d-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617183 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617195 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89464194-c498-4655-8868-a930de543542-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617209 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617223 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617237 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7mv\" (UniqueName: \"kubernetes.io/projected/bf49285b-876e-4a6d-8c3e-c8602687306d-kube-api-access-ms7mv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617249 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617262 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89464194-c498-4655-8868-a930de543542-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617274 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89464194-c498-4655-8868-a930de543542-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617286 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617300 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf49285b-876e-4a6d-8c3e-c8602687306d-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617312 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf49285b-876e-4a6d-8c3e-c8602687306d-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.617731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.617325 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e036dea4-c6a2-4956-a75c-8f5f0904d287-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.618398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.618363 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-socket" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.621019 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.620981 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-envoy" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:28.621426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.621401 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-token" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.621581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.621558 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-kube-api-access-swdbr" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "kube-api-access-swdbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:28.621656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.621588 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-podinfo" pod "e036dea4-c6a2-4956-a75c-8f5f0904d287" (UID: "e036dea4-c6a2-4956-a75c-8f5f0904d287"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:00:28.718810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.718727 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.718810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.718775 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdbr\" (UniqueName: \"kubernetes.io/projected/e036dea4-c6a2-4956-a75c-8f5f0904d287-kube-api-access-swdbr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.718810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.718786 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.718810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.718794 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.718810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.718804 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e036dea4-c6a2-4956-a75c-8f5f0904d287-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:28.848877 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.848842 4442 generic.go:356] "Generic (PLEG): container finished" podID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerID="a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4" exitCode=0 Oct 30 04:00:28.849320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.848896 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" Oct 30 04:00:28.849320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.848918 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" event={"ID":"e036dea4-c6a2-4956-a75c-8f5f0904d287","Type":"ContainerDied","Data":"a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4"} Oct 30 04:00:28.849320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.848966 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6" event={"ID":"e036dea4-c6a2-4956-a75c-8f5f0904d287","Type":"ContainerDied","Data":"7654aaa399e8eb22ea7d23c18ed80c984cb48ddc3f99f9f0b17aaf1856d6ec89"} Oct 30 04:00:28.849320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.848988 4442 scope.go:122] "RemoveContainer" containerID="a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4" Oct 30 04:00:28.850629 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.850602 4442 generic.go:356] "Generic (PLEG): container finished" podID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerID="f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1" exitCode=0 Oct 30 04:00:28.850898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.850661 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" Oct 30 04:00:28.850898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.850697 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" event={"ID":"bf49285b-876e-4a6d-8c3e-c8602687306d","Type":"ContainerDied","Data":"f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1"} Oct 30 04:00:28.850898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.850789 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25" event={"ID":"bf49285b-876e-4a6d-8c3e-c8602687306d","Type":"ContainerDied","Data":"f880c69cf5d6c5444bf323bed347c7d069c59e1203549b62e910e2b34c78cc5d"} Oct 30 04:00:28.855107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.855082 4442 generic.go:356] "Generic (PLEG): container finished" podID="89464194-c498-4655-8868-a930de543542" containerID="91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b" exitCode=0 Oct 30 04:00:28.855222 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.855181 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" Oct 30 04:00:28.855278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.855181 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" event={"ID":"89464194-c498-4655-8868-a930de543542","Type":"ContainerDied","Data":"91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b"} Oct 30 04:00:28.855337 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.855320 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2" event={"ID":"89464194-c498-4655-8868-a930de543542","Type":"ContainerDied","Data":"061cb29f246bb3d53e3f48c4b744d100d272c46a8ca71395af633e375bee4b62"} Oct 30 04:00:28.861102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.860993 4442 scope.go:122] "RemoveContainer" containerID="a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4" Oct 30 04:00:28.861444 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:28.861421 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4\": container with ID starting with a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4 not found: ID does not exist" containerID="a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4" Oct 30 04:00:28.861561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.861448 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4"} err="failed to get container status \"a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4\": rpc error: code = NotFound desc = could not find container \"a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4\": container with ID starting with a7deb303231965e92b9538f13341c6231e226747f3e31c29517ad2b4d24d81c4 not found: ID does not exist" Oct 30 04:00:28.861561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.861465 4442 scope.go:122] "RemoveContainer" containerID="f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1" Oct 30 04:00:28.871811 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.871795 4442 scope.go:122] "RemoveContainer" containerID="f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1" Oct 30 04:00:28.872077 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:28.872054 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1\": container with ID starting with f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1 not found: ID does not exist" containerID="f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1" Oct 30 04:00:28.872135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.872084 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1"} err="failed to get container status \"f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1\": rpc error: code = NotFound desc = could not find container \"f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1\": container with ID starting with f86e45f4f760dd3ad57a394eeb18088cad88c3b65c41d9506f3e527f026956a1 not found: ID does not exist" Oct 30 04:00:28.872135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.872101 4442 scope.go:122] "RemoveContainer" containerID="91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b" Oct 30 04:00:28.881455 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.881437 4442 scope.go:122] "RemoveContainer" containerID="91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b" Oct 30 04:00:28.881723 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:28.881707 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b\": container with ID starting with 91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b not found: ID does not exist" containerID="91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b" Oct 30 04:00:28.881802 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.881727 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b"} err="failed to get container status \"91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b\": rpc error: code = NotFound desc = could not find container \"91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b\": container with ID starting with 91b729574b3bbeb7c6d70817ab38726b9f3cccf87d3fa97e00f7a9e77d977a8b not found: ID does not exist" Oct 30 04:00:28.922806 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.922785 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2"] Oct 30 04:00:28.926789 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.926732 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-2vqg2"] Oct 30 04:00:28.963846 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.963809 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6"] Oct 30 04:00:28.967061 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.967035 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-95cl6"] Oct 30 04:00:28.995945 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.995866 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25"] Oct 30 04:00:28.999103 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:28.999084 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e0d09-openshift-default-db569dc47-qtr25"] Oct 30 04:00:29.307159 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:29.307122 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:30.109669 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:30.109638 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89464194-c498-4655-8868-a930de543542" path="/var/lib/kubelet/pods/89464194-c498-4655-8868-a930de543542/volumes" Oct 30 04:00:30.110245 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:30.110228 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" path="/var/lib/kubelet/pods/bf49285b-876e-4a6d-8c3e-c8602687306d/volumes" Oct 30 04:00:30.110703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:30.110689 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" path="/var/lib/kubelet/pods/e036dea4-c6a2-4956-a75c-8f5f0904d287/volumes" Oct 30 04:00:30.692656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:30.692622 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:31.863475 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:31.863444 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx"] Oct 30 04:00:31.864014 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:31.863687 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" podUID="4b8095a7-85ed-49b2-9676-c7c45e761463" containerName="maas-api" containerID="cri-o://e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001" gracePeriod=30 Oct 30 04:00:31.907956 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:31.907927 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:32.542451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.542419 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 04:00:32.608483 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.608451 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-maas-api-tls\") pod \"4b8095a7-85ed-49b2-9676-c7c45e761463\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " Oct 30 04:00:32.608673 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.608499 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8095a7-85ed-49b2-9676-c7c45e761463-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8095a7-85ed-49b2-9676-c7c45e761463-openshift-service-ca\") pod \"4b8095a7-85ed-49b2-9676-c7c45e761463\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " Oct 30 04:00:32.608673 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.608642 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8095a7-85ed-49b2-9676-c7c45e761463-kube-api-access-rxt6s\" (UniqueName: \"kubernetes.io/projected/4b8095a7-85ed-49b2-9676-c7c45e761463-kube-api-access-rxt6s\") pod \"4b8095a7-85ed-49b2-9676-c7c45e761463\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " Oct 30 04:00:32.608921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.608720 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-metrics-certs\") pod \"4b8095a7-85ed-49b2-9676-c7c45e761463\" (UID: \"4b8095a7-85ed-49b2-9676-c7c45e761463\") " Oct 30 04:00:32.609374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.609342 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8095a7-85ed-49b2-9676-c7c45e761463-openshift-service-ca" pod "4b8095a7-85ed-49b2-9676-c7c45e761463" (UID: "4b8095a7-85ed-49b2-9676-c7c45e761463"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:32.610265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.610238 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8095a7-85ed-49b2-9676-c7c45e761463-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:32.611430 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.611387 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-metrics-certs" pod "4b8095a7-85ed-49b2-9676-c7c45e761463" (UID: "4b8095a7-85ed-49b2-9676-c7c45e761463"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:32.611543 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.611463 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-maas-api-tls" pod "4b8095a7-85ed-49b2-9676-c7c45e761463" (UID: "4b8095a7-85ed-49b2-9676-c7c45e761463"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:00:32.612089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.612064 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8095a7-85ed-49b2-9676-c7c45e761463-kube-api-access-rxt6s" pod "4b8095a7-85ed-49b2-9676-c7c45e761463" (UID: "4b8095a7-85ed-49b2-9676-c7c45e761463"). InnerVolumeSpecName "kube-api-access-rxt6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:32.661896 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.661871 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:32.711794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.711721 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:32.711794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.711748 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxt6s\" (UniqueName: \"kubernetes.io/projected/4b8095a7-85ed-49b2-9676-c7c45e761463-kube-api-access-rxt6s\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:32.711794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.711789 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b8095a7-85ed-49b2-9676-c7c45e761463-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:32.888703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.888672 4442 generic.go:356] "Generic (PLEG): container finished" podID="4b8095a7-85ed-49b2-9676-c7c45e761463" containerID="e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001" exitCode=0 Oct 30 04:00:32.889124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.888739 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" Oct 30 04:00:32.889124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.888770 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" event={"ID":"4b8095a7-85ed-49b2-9676-c7c45e761463","Type":"ContainerDied","Data":"e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001"} Oct 30 04:00:32.889124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.888861 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx" event={"ID":"4b8095a7-85ed-49b2-9676-c7c45e761463","Type":"ContainerDied","Data":"09485e8ff64dd0d905ea0a4baaec5ef149e5aaa2134d98953ab549534b0b4e42"} Oct 30 04:00:32.889124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.888892 4442 scope.go:122] "RemoveContainer" containerID="e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001" Oct 30 04:00:32.901316 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.901300 4442 scope.go:122] "RemoveContainer" containerID="e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001" Oct 30 04:00:32.901621 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:32.901592 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001\": container with ID starting with e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001 not found: ID does not exist" containerID="e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001" Oct 30 04:00:32.901668 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.901629 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001"} err="failed to get container status \"e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001\": rpc error: code = NotFound desc = could not find container \"e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001\": container with ID starting with e0f2644b311f3f099f3ff4a83296e17b3028770613b1d64fa480c30572cd3001 not found: ID does not exist" Oct 30 04:00:32.937343 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.937317 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx"] Oct 30 04:00:32.947672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:32.947648 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a3d585-66cc7c76f7-zxxkx"] Oct 30 04:00:33.526533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.526463 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 04:00:33.623645 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.623608 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-credential-socket\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.623889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.623667 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-data\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.623889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.623807 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-envoy\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.624113 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624073 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-data" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:33.624199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624172 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-credential-socket" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:33.624301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624281 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-certs\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.624389 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624369 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff725a5-955f-433a-a175-019d98942e1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff725a5-955f-433a-a175-019d98942e1e-istiod-ca-cert\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.624568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624426 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-istio-token\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-istio-token\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.624568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624455 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-certs" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:33.624568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624496 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-socket\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.624568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624526 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dff725a5-955f-433a-a175-019d98942e1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff725a5-955f-433a-a175-019d98942e1e-istio-podinfo\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.624568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624565 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-kube-api-access-l84jn\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-kube-api-access-l84jn\") pod \"dff725a5-955f-433a-a175-019d98942e1e\" (UID: \"dff725a5-955f-433a-a175-019d98942e1e\") " Oct 30 04:00:33.624864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624805 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-socket" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:33.624996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.624972 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff725a5-955f-433a-a175-019d98942e1e-istiod-ca-cert" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:00:33.625437 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.625412 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.625547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.625440 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.625547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.625461 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.625547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.625475 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff725a5-955f-433a-a175-019d98942e1e-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.625547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.625489 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.626680 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.626654 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-envoy" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:00:33.626818 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.626799 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff725a5-955f-433a-a175-019d98942e1e-istio-podinfo" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:00:33.627007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.626985 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-istio-token" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:33.627172 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.627154 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-kube-api-access-l84jn" pod "dff725a5-955f-433a-a175-019d98942e1e" (UID: "dff725a5-955f-433a-a175-019d98942e1e"). InnerVolumeSpecName "kube-api-access-l84jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:00:33.726708 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.726666 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.726708 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.726698 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff725a5-955f-433a-a175-019d98942e1e-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.726708 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.726707 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84jn\" (UniqueName: \"kubernetes.io/projected/dff725a5-955f-433a-a175-019d98942e1e-kube-api-access-l84jn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.726708 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.726717 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff725a5-955f-433a-a175-019d98942e1e-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:00:33.896876 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.896788 4442 generic.go:356] "Generic (PLEG): container finished" podID="dff725a5-955f-433a-a175-019d98942e1e" containerID="a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8" exitCode=0 Oct 30 04:00:33.896876 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.896822 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" Oct 30 04:00:33.896876 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.896851 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" event={"ID":"dff725a5-955f-433a-a175-019d98942e1e","Type":"ContainerDied","Data":"a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8"} Oct 30 04:00:33.897495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.896903 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72" event={"ID":"dff725a5-955f-433a-a175-019d98942e1e","Type":"ContainerDied","Data":"0e339c0040d56c531ad87baa6a7bab6fa75cf3293c248007ef9aa8a8344b4882"} Oct 30 04:00:33.897495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.896931 4442 scope.go:122] "RemoveContainer" containerID="a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8" Oct 30 04:00:33.914562 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.914540 4442 scope.go:122] "RemoveContainer" containerID="a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8" Oct 30 04:00:33.914880 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:33.914864 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8\": container with ID starting with a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8 not found: ID does not exist" containerID="a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8" Oct 30 04:00:33.914941 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.914890 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8"} err="failed to get container status \"a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8\": rpc error: code = NotFound desc = could not find container \"a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8\": container with ID starting with a56a2c6a06fe717d8ddbb8508b1cd2f775a64312ca743f33e2f6da27f80604f8 not found: ID does not exist" Oct 30 04:00:33.947096 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.947073 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72"] Oct 30 04:00:33.958217 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:33.958196 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e10c0f-openshift-default-85f98fbc77-lmf72"] Oct 30 04:00:34.108627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:34.108596 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8095a7-85ed-49b2-9676-c7c45e761463" path="/var/lib/kubelet/pods/4b8095a7-85ed-49b2-9676-c7c45e761463/volumes" Oct 30 04:00:34.109106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:34.109091 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff725a5-955f-433a-a175-019d98942e1e" path="/var/lib/kubelet/pods/dff725a5-955f-433a-a175-019d98942e1e/volumes" Oct 30 04:00:34.187581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:34.187505 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:35.581313 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:35.581274 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:36.667922 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:36.667890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:36.770988 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:36.770255 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:37.108500 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:37.108470 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:37.381300 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:37.381220 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:37.579835 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:37.579799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:37.779555 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:37.779523 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:38.565910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:38.565880 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:38.902106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:38.902028 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:40.421695 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:40.421660 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:42.199564 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:42.199529 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:43.778366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:43.778333 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:44.437666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:44.437579 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:44.546291 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:44.546209 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" podUID="51e4a7b5-0531-459c-90e0-cb31b9caecd7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.218:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 04:00:44.549023 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:44.548961 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:44.564830 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:44.564393 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2"] Oct 30 04:00:44.640842 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:44.640808 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2"] Oct 30 04:00:44.676531 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:44.676499 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:44.796960 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:44.786002 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:46.303620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:46.303586 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:47.712476 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:47.712442 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:49.102043 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:49.102000 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:49.565296 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:49.565250 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" podUID="ef298974-3793-440a-ac80-95d7dc13c24a" containerName="payload-processing" containerID="cri-o://6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff" gracePeriod=25 Oct 30 04:00:49.647718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:49.647673 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" podUID="c811b25f-c31f-464c-8e3f-31c3ef046df4" containerName="payload-pre-processing" containerID="cri-o://f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727" gracePeriod=25 Oct 30 04:00:50.130817 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:50.130785 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:50.229012 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:50.228982 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:50.494381 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:50.494304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:50.627400 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:50.627369 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:50.726641 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:50.726609 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:51.656527 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:51.656494 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:51.906037 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:51.906007 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:52.592761 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:52.592715 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:53.426950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:53.426922 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:54.382565 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:54.382531 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:54.525814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:54.525780 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:54.621629 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:54.621598 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:54.863171 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:00:54.863133 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:00:55.236363 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:55.236277 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:55.651426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:55.651388 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" podUID="c811b25f-c31f-464c-8e3f-31c3ef046df4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:00:55.651857 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:55.651397 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" podUID="ef298974-3793-440a-ac80-95d7dc13c24a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:00:56.188479 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:56.188451 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:56.285443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:56.285408 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:56.896272 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:56.896237 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:58.383396 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:58.383356 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:59.792615 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:00:59.792578 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:01.192087 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:01.192052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:02.603883 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:02.603845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:04.112170 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:04.112141 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:05.563829 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:05.563795 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:05.660701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:05.660657 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" podUID="ef298974-3793-440a-ac80-95d7dc13c24a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:01:05.665779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:05.664290 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" podUID="c811b25f-c31f-464c-8e3f-31c3ef046df4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:01:07.073263 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:07.073222 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:08.551023 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:08.550993 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:09.951761 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:09.951722 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 04:01:09.988059 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:09.988024 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 04:01:10.001933 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.001908 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c811b25f-c31f-464c-8e3f-31c3ef046df4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c811b25f-c31f-464c-8e3f-31c3ef046df4-plugins-config-volume\") pod \"c811b25f-c31f-464c-8e3f-31c3ef046df4\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " Oct 30 04:01:10.002102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.002085 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c811b25f-c31f-464c-8e3f-31c3ef046df4-kube-api-access-c28s8\" (UniqueName: \"kubernetes.io/projected/c811b25f-c31f-464c-8e3f-31c3ef046df4-kube-api-access-c28s8\") pod \"c811b25f-c31f-464c-8e3f-31c3ef046df4\" (UID: \"c811b25f-c31f-464c-8e3f-31c3ef046df4\") " Oct 30 04:01:10.002173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.002129 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef298974-3793-440a-ac80-95d7dc13c24a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef298974-3793-440a-ac80-95d7dc13c24a-plugins-config-volume\") pod \"ef298974-3793-440a-ac80-95d7dc13c24a\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " Oct 30 04:01:10.002173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.002154 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef298974-3793-440a-ac80-95d7dc13c24a-kube-api-access-kgq5v\" (UniqueName: \"kubernetes.io/projected/ef298974-3793-440a-ac80-95d7dc13c24a-kube-api-access-kgq5v\") pod \"ef298974-3793-440a-ac80-95d7dc13c24a\" (UID: \"ef298974-3793-440a-ac80-95d7dc13c24a\") " Oct 30 04:01:10.002337 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.002311 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c811b25f-c31f-464c-8e3f-31c3ef046df4-plugins-config-volume" pod "c811b25f-c31f-464c-8e3f-31c3ef046df4" (UID: "c811b25f-c31f-464c-8e3f-31c3ef046df4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:01:10.003045 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.002602 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef298974-3793-440a-ac80-95d7dc13c24a-plugins-config-volume" pod "ef298974-3793-440a-ac80-95d7dc13c24a" (UID: "ef298974-3793-440a-ac80-95d7dc13c24a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:01:10.003045 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.002921 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c811b25f-c31f-464c-8e3f-31c3ef046df4-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:10.003045 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.002942 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef298974-3793-440a-ac80-95d7dc13c24a-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:10.005270 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.005242 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c811b25f-c31f-464c-8e3f-31c3ef046df4-kube-api-access-c28s8" pod "c811b25f-c31f-464c-8e3f-31c3ef046df4" (UID: "c811b25f-c31f-464c-8e3f-31c3ef046df4"). InnerVolumeSpecName "kube-api-access-c28s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:10.007412 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.007387 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef298974-3793-440a-ac80-95d7dc13c24a-kube-api-access-kgq5v" pod "ef298974-3793-440a-ac80-95d7dc13c24a" (UID: "ef298974-3793-440a-ac80-95d7dc13c24a"). InnerVolumeSpecName "kube-api-access-kgq5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:10.103975 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.103945 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c28s8\" (UniqueName: \"kubernetes.io/projected/c811b25f-c31f-464c-8e3f-31c3ef046df4-kube-api-access-c28s8\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:10.103975 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.103973 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgq5v\" (UniqueName: \"kubernetes.io/projected/ef298974-3793-440a-ac80-95d7dc13c24a-kube-api-access-kgq5v\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:10.124135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.124111 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:01:10.194098 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.194037 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:10.365171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.365132 4442 generic.go:356] "Generic (PLEG): container finished" podID="ef298974-3793-440a-ac80-95d7dc13c24a" containerID="6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff" exitCode=0 Oct 30 04:01:10.365381 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.365186 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" Oct 30 04:01:10.365381 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.365216 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" event={"ID":"ef298974-3793-440a-ac80-95d7dc13c24a","Type":"ContainerDied","Data":"6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff"} Oct 30 04:01:10.365381 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.365299 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2" event={"ID":"ef298974-3793-440a-ac80-95d7dc13c24a","Type":"ContainerDied","Data":"0f9e1e2a4620975a34be33ec70e70833d49d88c195cad7b61225e3998136d528"} Oct 30 04:01:10.365381 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.365330 4442 scope.go:122] "RemoveContainer" containerID="6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff" Oct 30 04:01:10.366797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.366775 4442 generic.go:356] "Generic (PLEG): container finished" podID="c811b25f-c31f-464c-8e3f-31c3ef046df4" containerID="f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727" exitCode=0 Oct 30 04:01:10.366940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.366845 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" event={"ID":"c811b25f-c31f-464c-8e3f-31c3ef046df4","Type":"ContainerDied","Data":"f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727"} Oct 30 04:01:10.366940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.366881 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" Oct 30 04:01:10.367078 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.366887 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2" event={"ID":"c811b25f-c31f-464c-8e3f-31c3ef046df4","Type":"ContainerDied","Data":"ef7a2f7efb13d1e51c3363f53c2c75c99f6df89f09bf9ba0899d6db2dfb4d881"} Oct 30 04:01:10.376225 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.376208 4442 scope.go:122] "RemoveContainer" containerID="6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff" Oct 30 04:01:10.376471 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:01:10.376449 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff\": container with ID starting with 6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff not found: ID does not exist" containerID="6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff" Oct 30 04:01:10.376561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.376477 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff"} err="failed to get container status \"6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff\": rpc error: code = NotFound desc = could not find container \"6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff\": container with ID starting with 6df448e069018e2112eabb17b4ff9c8a2171a440fdb3a03c132b6fdc60a212ff not found: ID does not exist" Oct 30 04:01:10.376561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.376493 4442 scope.go:122] "RemoveContainer" containerID="f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727" Oct 30 04:01:10.388905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.388884 4442 scope.go:122] "RemoveContainer" containerID="f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727" Oct 30 04:01:10.389286 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:01:10.389266 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727\": container with ID starting with f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727 not found: ID does not exist" containerID="f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727" Oct 30 04:01:10.389377 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.389293 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727"} err="failed to get container status \"f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727\": rpc error: code = NotFound desc = could not find container \"f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727\": container with ID starting with f5bae87f7babd7b4a16bb4631de0387d3843ecad6a2839012756a17431e26727 not found: ID does not exist" Oct 30 04:01:10.414042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.414019 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2"] Oct 30 04:01:10.417746 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.417726 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4206d9-6f8784957f-kqhr2"] Oct 30 04:01:10.450508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.450434 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2"] Oct 30 04:01:10.454818 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:10.454792 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4206d9-c8d98bbb8-jbsv2"] Oct 30 04:01:11.895321 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:11.895284 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:12.168826 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:12.161398 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c811b25f-c31f-464c-8e3f-31c3ef046df4" path="/var/lib/kubelet/pods/c811b25f-c31f-464c-8e3f-31c3ef046df4/volumes" Oct 30 04:01:12.168826 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:12.162059 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef298974-3793-440a-ac80-95d7dc13c24a" path="/var/lib/kubelet/pods/ef298974-3793-440a-ac80-95d7dc13c24a/volumes" Oct 30 04:01:13.957089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:13.955380 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:15.406996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:15.406958 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:16.591105 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:16.591070 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f"] Oct 30 04:01:16.614620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:16.614590 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:17.210692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:17.210636 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:18.578131 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:18.578078 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:19.998214 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:19.998177 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:21.494974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:21.494944 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:22.909682 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:22.909645 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:24.108147 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:24.108104 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:24.860581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:24.860533 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:24.862854 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:01:24.862826 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:01:26.371357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:26.371319 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:27.897007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:27.896973 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:29.467605 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:29.467573 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:30.983696 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:30.983661 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:31.616265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:31.616215 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" podUID="51e4a7b5-0531-459c-90e0-cb31b9caecd7" containerName="main" containerID="cri-o://cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139" gracePeriod=45 Oct 30 04:01:31.884102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:31.884046 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 04:01:32.021747 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.021702 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-home\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-home\") pod \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " Oct 30 04:01:32.022273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.021815 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tmp-dir\") pod \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " Oct 30 04:01:32.022273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.021897 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-model-cache\") pod \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " Oct 30 04:01:32.022273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022049 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tls-certs\") pod \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " Oct 30 04:01:32.022273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022090 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-home" pod "51e4a7b5-0531-459c-90e0-cb31b9caecd7" (UID: "51e4a7b5-0531-459c-90e0-cb31b9caecd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:32.022273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022107 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tmp-dir" pod "51e4a7b5-0531-459c-90e0-cb31b9caecd7" (UID: "51e4a7b5-0531-459c-90e0-cb31b9caecd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:32.022273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022123 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e4a7b5-0531-459c-90e0-cb31b9caecd7-kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/51e4a7b5-0531-459c-90e0-cb31b9caecd7-kube-api-access-sbvmq\") pod \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " Oct 30 04:01:32.022273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022167 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-dshm\") pod \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\" (UID: \"51e4a7b5-0531-459c-90e0-cb31b9caecd7\") " Oct 30 04:01:32.022687 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022476 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-model-cache" pod "51e4a7b5-0531-459c-90e0-cb31b9caecd7" (UID: "51e4a7b5-0531-459c-90e0-cb31b9caecd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:32.022950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022876 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:32.022950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022925 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:32.022950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.022940 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:32.024818 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.024774 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tls-certs" pod "51e4a7b5-0531-459c-90e0-cb31b9caecd7" (UID: "51e4a7b5-0531-459c-90e0-cb31b9caecd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:01:32.024818 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.024784 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-dshm" pod "51e4a7b5-0531-459c-90e0-cb31b9caecd7" (UID: "51e4a7b5-0531-459c-90e0-cb31b9caecd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:32.024818 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.024790 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e4a7b5-0531-459c-90e0-cb31b9caecd7-kube-api-access-sbvmq" pod "51e4a7b5-0531-459c-90e0-cb31b9caecd7" (UID: "51e4a7b5-0531-459c-90e0-cb31b9caecd7"). InnerVolumeSpecName "kube-api-access-sbvmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:32.123828 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.123799 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e4a7b5-0531-459c-90e0-cb31b9caecd7-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:32.123828 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.123827 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvmq\" (UniqueName: \"kubernetes.io/projected/51e4a7b5-0531-459c-90e0-cb31b9caecd7-kube-api-access-sbvmq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:32.123993 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.123836 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e4a7b5-0531-459c-90e0-cb31b9caecd7-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:32.203158 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.203077 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:32.335125 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.335090 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:32.525873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.525822 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh"] Oct 30 04:01:32.528953 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.528919 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:32.623495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.623391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:32.695546 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.695510 4442 generic.go:356] "Generic (PLEG): container finished" podID="51e4a7b5-0531-459c-90e0-cb31b9caecd7" containerID="cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139" exitCode=0 Oct 30 04:01:32.696278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.696254 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" event={"ID":"51e4a7b5-0531-459c-90e0-cb31b9caecd7","Type":"ContainerDied","Data":"cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139"} Oct 30 04:01:32.696569 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.696553 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" event={"ID":"51e4a7b5-0531-459c-90e0-cb31b9caecd7","Type":"ContainerDied","Data":"0ee9e74b26db413a5e45ead7835429cad0c68068af9d88b83fdb7bf4dd153de2"} Oct 30 04:01:32.696690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.696678 4442 scope.go:122] "RemoveContainer" containerID="cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139" Oct 30 04:01:32.696909 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.696515 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f" Oct 30 04:01:32.732508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.732384 4442 scope.go:122] "RemoveContainer" containerID="cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139" Oct 30 04:01:32.733102 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:01:32.733023 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139\": container with ID starting with cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139 not found: ID does not exist" containerID="cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139" Oct 30 04:01:32.733102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.733061 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139"} err="failed to get container status \"cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139\": rpc error: code = NotFound desc = could not find container \"cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139\": container with ID starting with cd8e90e91304e18da9925c867b101a8316c18f1ceb9f5b7b3eaf5ef68cc49139 not found: ID does not exist" Oct 30 04:01:32.758759 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.758718 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f"] Oct 30 04:01:32.777218 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:32.777149 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0393ed/test-model-0393ed-kserve-864d548475-qqf2f"] Oct 30 04:01:33.008508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:33.008467 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:33.110660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:33.110583 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:33.373531 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:33.373439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:34.109222 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:34.109188 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e4a7b5-0531-459c-90e0-cb31b9caecd7" path="/var/lib/kubelet/pods/51e4a7b5-0531-459c-90e0-cb31b9caecd7/volumes" Oct 30 04:01:34.380732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:34.380639 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:34.472708 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:34.472675 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:34.571235 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:34.571185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:34.679715 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:34.679633 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:35.177950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:35.177907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:35.316792 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:35.316733 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:35.559351 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:35.559297 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:36.186976 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:36.186935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:37.724874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:37.724835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:38.322884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:38.322853 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf"] Oct 30 04:01:38.433349 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:38.433314 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:38.556666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:38.553576 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:39.291257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:39.291227 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:39.497136 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:39.497102 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:40.903096 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:40.903065 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:41.581392 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:41.581364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:42.144558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:42.144523 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:42.251043 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:42.251013 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:42.519244 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:42.519214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:43.323227 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:43.323188 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" podUID="f415ec38-1e43-40b9-868d-034586bad482" containerName="payload-pre-processing" containerID="cri-o://00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208" gracePeriod=25 Oct 30 04:01:43.983618 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:43.983586 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:45.411892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:45.411862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:45.527783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:45.526177 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:46.895483 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:46.895456 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:47.551048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.551012 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" podUID="2aeabff1-b68a-47b1-8010-ef639d3bf14c" containerName="main" containerID="cri-o://78bad4c9ca9e980a8a6ea23d3d8bad059db1701d5de101f54df17754620d73aa" gracePeriod=45 Oct 30 04:01:47.826218 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.826189 4442 generic.go:356] "Generic (PLEG): container finished" podID="2aeabff1-b68a-47b1-8010-ef639d3bf14c" containerID="78bad4c9ca9e980a8a6ea23d3d8bad059db1701d5de101f54df17754620d73aa" exitCode=0 Oct 30 04:01:47.826371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.826270 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" event={"ID":"2aeabff1-b68a-47b1-8010-ef639d3bf14c","Type":"ContainerDied","Data":"78bad4c9ca9e980a8a6ea23d3d8bad059db1701d5de101f54df17754620d73aa"} Oct 30 04:01:47.826371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.826325 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" event={"ID":"2aeabff1-b68a-47b1-8010-ef639d3bf14c","Type":"ContainerDied","Data":"eb66b5f426615fe3d65d28c891452e812088bb90a17d2a4aa87ffeede905473b"} Oct 30 04:01:47.826371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.826340 4442 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb66b5f426615fe3d65d28c891452e812088bb90a17d2a4aa87ffeede905473b" Oct 30 04:01:47.836698 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.836680 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 04:01:47.924286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924260 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aeabff1-b68a-47b1-8010-ef639d3bf14c-kube-api-access-kjl29\" (UniqueName: \"kubernetes.io/projected/2aeabff1-b68a-47b1-8010-ef639d3bf14c-kube-api-access-kjl29\") pod \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " Oct 30 04:01:47.924693 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924369 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tls-certs\") pod \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " Oct 30 04:01:47.924693 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924429 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-dshm\") pod \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " Oct 30 04:01:47.924693 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924490 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tmp-dir\") pod \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " Oct 30 04:01:47.924693 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924516 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-home\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-home\") pod \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " Oct 30 04:01:47.924693 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924593 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-model-cache\") pod \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\" (UID: \"2aeabff1-b68a-47b1-8010-ef639d3bf14c\") " Oct 30 04:01:47.925024 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924794 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tmp-dir" pod "2aeabff1-b68a-47b1-8010-ef639d3bf14c" (UID: "2aeabff1-b68a-47b1-8010-ef639d3bf14c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:47.925024 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924859 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-home" pod "2aeabff1-b68a-47b1-8010-ef639d3bf14c" (UID: "2aeabff1-b68a-47b1-8010-ef639d3bf14c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:47.925024 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.924970 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-model-cache" pod "2aeabff1-b68a-47b1-8010-ef639d3bf14c" (UID: "2aeabff1-b68a-47b1-8010-ef639d3bf14c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:47.925479 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.925453 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:47.925563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.925483 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:47.925563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.925498 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:47.926925 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.926900 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tls-certs" pod "2aeabff1-b68a-47b1-8010-ef639d3bf14c" (UID: "2aeabff1-b68a-47b1-8010-ef639d3bf14c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:01:47.927026 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.926936 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-dshm" pod "2aeabff1-b68a-47b1-8010-ef639d3bf14c" (UID: "2aeabff1-b68a-47b1-8010-ef639d3bf14c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:47.927026 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:47.926972 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aeabff1-b68a-47b1-8010-ef639d3bf14c-kube-api-access-kjl29" pod "2aeabff1-b68a-47b1-8010-ef639d3bf14c" (UID: "2aeabff1-b68a-47b1-8010-ef639d3bf14c"). InnerVolumeSpecName "kube-api-access-kjl29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:48.026551 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.026520 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjl29\" (UniqueName: \"kubernetes.io/projected/2aeabff1-b68a-47b1-8010-ef639d3bf14c-kube-api-access-kjl29\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:48.026551 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.026549 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeabff1-b68a-47b1-8010-ef639d3bf14c-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:48.026794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.026559 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2aeabff1-b68a-47b1-8010-ef639d3bf14c-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:01:48.383075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.382977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:48.835574 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.835545 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh" Oct 30 04:01:48.860721 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.860674 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:48.889449 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.889419 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh"] Oct 30 04:01:48.900266 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:48.900239 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b267f8/test-model-b267f8-kserve-85fd48cc49-jzmvh"] Oct 30 04:01:49.902506 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:49.902470 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:50.109385 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:50.109348 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aeabff1-b68a-47b1-8010-ef639d3bf14c" path="/var/lib/kubelet/pods/2aeabff1-b68a-47b1-8010-ef639d3bf14c/volumes" Oct 30 04:01:50.570321 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:50.570281 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" podUID="f415ec38-1e43-40b9-868d-034586bad482" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:01:51.338715 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:51.338670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:51.776454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:51.776412 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:52.712221 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:52.712183 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:52.812735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:52.812705 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:54.286104 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:54.286073 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:54.549952 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:54.549868 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:54.861658 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:01:54.861573 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:01:55.684741 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:55.684709 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:56.643193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:56.643158 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:01:57.317781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:57.313796 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:58.909960 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:01:58.909925 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:00.099935 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:00.099898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:00.570702 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:00.570657 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" podUID="f415ec38-1e43-40b9-868d-034586bad482" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:02:00.767961 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:00.767926 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:00.844869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:00.844784 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:02.218420 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:02.218387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:03.608842 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.608811 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 04:02:03.752456 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.752420 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f415ec38-1e43-40b9-868d-034586bad482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f415ec38-1e43-40b9-868d-034586bad482-plugins-config-volume\") pod \"f415ec38-1e43-40b9-868d-034586bad482\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " Oct 30 04:02:03.752659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.752479 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f415ec38-1e43-40b9-868d-034586bad482-kube-api-access-9284g\" (UniqueName: \"kubernetes.io/projected/f415ec38-1e43-40b9-868d-034586bad482-kube-api-access-9284g\") pod \"f415ec38-1e43-40b9-868d-034586bad482\" (UID: \"f415ec38-1e43-40b9-868d-034586bad482\") " Oct 30 04:02:03.752902 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.752879 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f415ec38-1e43-40b9-868d-034586bad482-plugins-config-volume" pod "f415ec38-1e43-40b9-868d-034586bad482" (UID: "f415ec38-1e43-40b9-868d-034586bad482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:03.754783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.754724 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f415ec38-1e43-40b9-868d-034586bad482-kube-api-access-9284g" pod "f415ec38-1e43-40b9-868d-034586bad482" (UID: "f415ec38-1e43-40b9-868d-034586bad482"). InnerVolumeSpecName "kube-api-access-9284g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:03.854564 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.854497 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f415ec38-1e43-40b9-868d-034586bad482-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:03.854564 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.854525 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9284g\" (UniqueName: \"kubernetes.io/projected/f415ec38-1e43-40b9-868d-034586bad482-kube-api-access-9284g\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:03.885122 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.885098 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:03.997603 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.997565 4442 generic.go:356] "Generic (PLEG): container finished" podID="f415ec38-1e43-40b9-868d-034586bad482" containerID="00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208" exitCode=0 Oct 30 04:02:04.000873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.997895 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" event={"ID":"f415ec38-1e43-40b9-868d-034586bad482","Type":"ContainerDied","Data":"00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208"} Oct 30 04:02:04.000873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.997957 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" event={"ID":"f415ec38-1e43-40b9-868d-034586bad482","Type":"ContainerDied","Data":"f39118eedf795fb8136ce26ce87e1a919be9b27eedac4fd4661f4a32214c12e0"} Oct 30 04:02:04.000873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.997991 4442 scope.go:122] "RemoveContainer" containerID="00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208" Oct 30 04:02:04.000873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:03.998151 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf" Oct 30 04:02:04.012967 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:04.012942 4442 scope.go:122] "RemoveContainer" containerID="00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208" Oct 30 04:02:04.013330 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:04.013296 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208\": container with ID starting with 00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208 not found: ID does not exist" containerID="00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208" Oct 30 04:02:04.013408 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:04.013341 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208"} err="failed to get container status \"00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208\": rpc error: code = NotFound desc = could not find container \"00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208\": container with ID starting with 00526bee4f6f6d8b8c4f97ecf710e86ce7671d2dec6a3695f9e4d64d8e1f3208 not found: ID does not exist" Oct 30 04:02:04.049253 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:04.049229 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf"] Oct 30 04:02:04.052467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:04.052444 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0393ed-64994f786c-cfxsf"] Oct 30 04:02:04.109393 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:04.109333 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f415ec38-1e43-40b9-868d-034586bad482" path="/var/lib/kubelet/pods/f415ec38-1e43-40b9-868d-034586bad482/volumes" Oct 30 04:02:05.196780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:05.196725 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx"] Oct 30 04:02:05.197232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:05.196944 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" containerID="cri-o://4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542" gracePeriod=30 Oct 30 04:02:05.293922 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:05.293887 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:06.762695 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:06.762663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:08.209899 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:08.209864 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:08.858523 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:08.858487 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:08.957021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:08.956968 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:09.516356 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.516323 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:09.616714 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.616678 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:09.644212 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644179 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9"] Oct 30 04:02:09.644872 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644844 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:09.644872 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644860 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:09.644872 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644873 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644881 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644895 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644900 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644914 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f415ec38-1e43-40b9-868d-034586bad482" containerName="payload-pre-processing" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644918 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f415ec38-1e43-40b9-868d-034586bad482" containerName="payload-pre-processing" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644929 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644935 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644947 4442 cpu_manager.go:414] "containerMap: removing container" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644954 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644968 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644975 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644985 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.644991 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645003 4442 cpu_manager.go:414] "containerMap: removing container" podUID="51e4a7b5-0531-459c-90e0-cb31b9caecd7" containerName="main" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645009 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e4a7b5-0531-459c-90e0-cb31b9caecd7" containerName="main" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645022 4442 cpu_manager.go:414] "containerMap: removing container" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645029 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645038 4442 cpu_manager.go:414] "containerMap: removing container" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645043 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645062 4442 cpu_manager.go:414] "containerMap: removing container" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645069 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645080 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645087 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645097 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645104 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645116 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" Oct 30 04:02:09.645109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645123 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645135 4442 cpu_manager.go:414] "containerMap: removing container" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645142 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645162 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645169 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645183 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4b8095a7-85ed-49b2-9676-c7c45e761463" containerName="maas-api" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645189 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8095a7-85ed-49b2-9676-c7c45e761463" containerName="maas-api" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645200 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ef298974-3793-440a-ac80-95d7dc13c24a" containerName="payload-processing" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645206 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef298974-3793-440a-ac80-95d7dc13c24a" containerName="payload-processing" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645217 4442 cpu_manager.go:414] "containerMap: removing container" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645223 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645238 4442 cpu_manager.go:414] "containerMap: removing container" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645244 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645253 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645259 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645279 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645287 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645300 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c811b25f-c31f-464c-8e3f-31c3ef046df4" containerName="payload-pre-processing" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645307 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c811b25f-c31f-464c-8e3f-31c3ef046df4" containerName="payload-pre-processing" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645323 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645329 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645338 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2aeabff1-b68a-47b1-8010-ef639d3bf14c" containerName="main" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645344 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aeabff1-b68a-47b1-8010-ef639d3bf14c" containerName="main" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645497 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645506 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645512 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645518 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf49285b-876e-4a6d-8c3e-c8602687306d" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645524 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645531 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff725a5-955f-433a-a175-019d98942e1e" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645536 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8095a7-85ed-49b2-9676-c7c45e761463" containerName="maas-api" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645544 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef298974-3793-440a-ac80-95d7dc13c24a" containerName="payload-processing" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645549 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="89464194-c498-4655-8868-a930de543542" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645553 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645558 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645564 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645572 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e036dea4-c6a2-4956-a75c-8f5f0904d287" containerName="istio-proxy" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645579 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c811b25f-c31f-464c-8e3f-31c3ef046df4" containerName="payload-pre-processing" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645585 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645590 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aeabff1-b68a-47b1-8010-ef639d3bf14c" containerName="main" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645596 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645602 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645608 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f415ec38-1e43-40b9-868d-034586bad482" containerName="payload-pre-processing" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645615 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645620 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e4a7b5-0531-459c-90e0-cb31b9caecd7" containerName="main" Oct 30 04:02:09.646048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.645624 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:09.650506 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.650486 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.669428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.669405 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9"] Oct 30 04:02:09.677656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.677605 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"default-dockercfg-v8mlq\"" Oct 30 04:02:09.678557 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.677910 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"openshift-service-ca.crt\"" Oct 30 04:02:09.678557 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.678028 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"kube-root-ca.crt\"" Oct 30 04:02:09.679560 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.678182 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e1a0c2\"/\"sub-test-model-e1a0c2-kserve-self-signed-certs\"" Oct 30 04:02:09.746448 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.746418 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4nvt\" (UniqueName: \"kubernetes.io/projected/c80b1e04-c613-49cf-944f-b46f80aa88cf-kube-api-access-f4nvt\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.746655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.746458 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-dshm\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.746655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.746612 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-model-cache\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.746855 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.746672 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80b1e04-c613-49cf-944f-b46f80aa88cf-tls-certs\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.746855 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.746785 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-home\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.746855 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.746825 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-tmp-dir\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.853029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.852932 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4nvt\" (UniqueName: \"kubernetes.io/projected/c80b1e04-c613-49cf-944f-b46f80aa88cf-kube-api-access-f4nvt\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.853303 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.853267 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-dshm\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.854004 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.853966 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-model-cache\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.854237 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.854206 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80b1e04-c613-49cf-944f-b46f80aa88cf-tls-certs\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.854445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.854429 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-home\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.862780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.854592 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-tmp-dir\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.864163 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.864120 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-model-cache\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.864591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.864568 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-tmp-dir\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.865459 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.865439 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-home\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.877133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.877050 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-dshm\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.945389 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.940663 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749"] Oct 30 04:02:09.945389 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.940942 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" containerID="cri-o://9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d" gracePeriod=30 Oct 30 04:02:09.950599 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.945983 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80b1e04-c613-49cf-944f-b46f80aa88cf-tls-certs\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.950599 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.950105 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4nvt\" (UniqueName: \"kubernetes.io/projected/c80b1e04-c613-49cf-944f-b46f80aa88cf-kube-api-access-f4nvt\") pod \"sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:09.951564 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.951455 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:09.985133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:09.984971 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:10.188943 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.188841 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:10.455326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.455269 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 04:02:10.456908 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.456882 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9"] Oct 30 04:02:10.457306 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:02:10.457284 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc80b1e04_c613_49cf_944f_b46f80aa88cf.slice/crio-392cc7aab5f2b7f2d807950487a1b974c888f27686cbd16fe31b600c2af23896 WatchSource:0}: Error finding container 392cc7aab5f2b7f2d807950487a1b974c888f27686cbd16fe31b600c2af23896: Status 404 returned error can't find the container with id 392cc7aab5f2b7f2d807950487a1b974c888f27686cbd16fe31b600c2af23896 Oct 30 04:02:10.519177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519143 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-credential-socket\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519200 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-kube-api-access-vg5jc\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-kube-api-access-vg5jc\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519241 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60232eb-dc73-4290-a115-df93a7b1c6f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60232eb-dc73-4290-a115-df93a7b1c6f8-istiod-ca-cert\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519279 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-certs\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519397 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-data\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519418 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-credential-socket" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:10.519592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519556 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-podinfo\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519616 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-certs" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:10.519972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519645 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-token\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-token\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519735 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-socket\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.519972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519658 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60232eb-dc73-4290-a115-df93a7b1c6f8-istiod-ca-cert" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:10.519972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519808 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-data" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:10.519972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.519817 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-envoy\") pod \"d60232eb-dc73-4290-a115-df93a7b1c6f8\" (UID: \"d60232eb-dc73-4290-a115-df93a7b1c6f8\") " Oct 30 04:02:10.520602 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.520277 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-socket" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:10.520602 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.520523 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.520602 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.520544 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d60232eb-dc73-4290-a115-df93a7b1c6f8-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.520602 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.520559 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.520602 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.520572 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.520907 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.520648 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.522489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.522460 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-envoy" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:10.522685 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.522659 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-token" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:10.523143 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.523124 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-podinfo" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:02:10.524582 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.524558 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-kube-api-access-vg5jc" pod "d60232eb-dc73-4290-a115-df93a7b1c6f8" (UID: "d60232eb-dc73-4290-a115-df93a7b1c6f8"). InnerVolumeSpecName "kube-api-access-vg5jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:10.621656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.621622 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.621656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.621657 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.621656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.621666 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d60232eb-dc73-4290-a115-df93a7b1c6f8-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.621942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.621675 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg5jc\" (UniqueName: \"kubernetes.io/projected/d60232eb-dc73-4290-a115-df93a7b1c6f8-kube-api-access-vg5jc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:10.731517 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:10.731426 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:11.199999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.199969 4442 generic.go:356] "Generic (PLEG): container finished" podID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerID="4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542" exitCode=0 Oct 30 04:02:11.200168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.200015 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" Oct 30 04:02:11.200168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.200048 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" event={"ID":"d60232eb-dc73-4290-a115-df93a7b1c6f8","Type":"ContainerDied","Data":"4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542"} Oct 30 04:02:11.200168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.200101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx" event={"ID":"d60232eb-dc73-4290-a115-df93a7b1c6f8","Type":"ContainerDied","Data":"3786203bf6a7235e0162ace87f7743b0cab52411461d426b608cff9f3afcb4f3"} Oct 30 04:02:11.200168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.200122 4442 scope.go:122] "RemoveContainer" containerID="4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542" Oct 30 04:02:11.202169 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.202147 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" event={"ID":"c80b1e04-c613-49cf-944f-b46f80aa88cf","Type":"ContainerStarted","Data":"32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c"} Oct 30 04:02:11.202285 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.202180 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" event={"ID":"c80b1e04-c613-49cf-944f-b46f80aa88cf","Type":"ContainerStarted","Data":"392cc7aab5f2b7f2d807950487a1b974c888f27686cbd16fe31b600c2af23896"} Oct 30 04:02:11.211305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.211288 4442 scope.go:122] "RemoveContainer" containerID="4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542" Oct 30 04:02:11.211577 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:11.211556 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542\": container with ID starting with 4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542 not found: ID does not exist" containerID="4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542" Oct 30 04:02:11.211675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.211581 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542"} err="failed to get container status \"4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542\": rpc error: code = NotFound desc = could not find container \"4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542\": container with ID starting with 4fca4602b8b8496a22c892952d27a28214b9f8fe9449056982ffa97f321fb542 not found: ID does not exist" Oct 30 04:02:11.261852 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.261831 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx"] Oct 30 04:02:11.272822 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.272800 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-398c25-openshift-default-758b69476f-9bqqx"] Oct 30 04:02:11.303036 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.302993 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" podStartSLOduration=2.302981592 podStartE2EDuration="2.302981592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:11.300117956 +0000 UTC m=+3555.302471144" watchObservedRunningTime="2026-10-30 04:02:11.302981592 +0000 UTC m=+3555.305334822" Oct 30 04:02:11.757554 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.757517 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:11.985973 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:11.985945 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:12.108809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:12.108686 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" path="/var/lib/kubelet/pods/d60232eb-dc73-4290-a115-df93a7b1c6f8/volumes" Oct 30 04:02:13.220573 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:13.220542 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:14.694235 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:14.694193 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:15.220461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.220407 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 04:02:15.231431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.231401 4442 generic.go:356] "Generic (PLEG): container finished" podID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerID="9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d" exitCode=0 Oct 30 04:02:15.231561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.231447 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" Oct 30 04:02:15.231561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.231479 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" event={"ID":"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b","Type":"ContainerDied","Data":"9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d"} Oct 30 04:02:15.231561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.231532 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749" event={"ID":"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b","Type":"ContainerDied","Data":"1671cea5726b5f66b4c3b1bf44563316ab4a4de122a7860b6e2730a90d7aa767"} Oct 30 04:02:15.231561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.231559 4442 scope.go:122] "RemoveContainer" containerID="9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d" Oct 30 04:02:15.241915 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.241895 4442 scope.go:122] "RemoveContainer" containerID="8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9" Oct 30 04:02:15.251635 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.251618 4442 scope.go:122] "RemoveContainer" containerID="9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d" Oct 30 04:02:15.251970 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:15.251950 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d\": container with ID starting with 9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d not found: ID does not exist" containerID="9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d" Oct 30 04:02:15.252044 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.251974 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d"} err="failed to get container status \"9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d\": rpc error: code = NotFound desc = could not find container \"9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d\": container with ID starting with 9677b787f99f8c6c3fc3b8e1f4826e7656d07a5ad5cf42d0e505cf45ce67229d not found: ID does not exist" Oct 30 04:02:15.252044 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.251992 4442 scope.go:122] "RemoveContainer" containerID="8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9" Oct 30 04:02:15.252272 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:15.252256 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9\": container with ID starting with 8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9 not found: ID does not exist" containerID="8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9" Oct 30 04:02:15.252311 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.252281 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9"} err="failed to get container status \"8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9\": rpc error: code = NotFound desc = could not find container \"8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9\": container with ID starting with 8afa73557c62e8673f008a2c42ccac4a4d5fceccf691663e58c39f948b3ac3a9 not found: ID does not exist" Oct 30 04:02:15.297567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297539 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-certs\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.297779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297639 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-token\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-token\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.297779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297666 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-kube-api-access-wmfdm\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-kube-api-access-wmfdm\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.297779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297686 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-credential-socket\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.297999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297826 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-socket\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.297999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297888 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-data\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.297999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297910 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-certs" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:15.297999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.297954 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-envoy\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.298220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298017 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istiod-ca-cert\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.298220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298020 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-credential-socket" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:15.298220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298062 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-podinfo\") pod \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\" (UID: \"5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b\") " Oct 30 04:02:15.298220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298120 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-socket" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:15.298220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298200 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-data" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:15.298495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298434 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istiod-ca-cert" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:15.298660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298642 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.298768 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298666 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.298768 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298682 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.298768 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298710 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.298768 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.298723 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.300450 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.300430 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-token" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:15.300806 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.300785 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-kube-api-access-wmfdm" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "kube-api-access-wmfdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:15.300911 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.300893 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-envoy" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:15.300911 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.300898 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-podinfo" pod "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" (UID: "5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:02:15.399626 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.399596 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.399626 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.399625 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.399626 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.399634 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmfdm\" (UniqueName: \"kubernetes.io/projected/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-kube-api-access-wmfdm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.399892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.399643 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:15.576576 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.576542 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749"] Oct 30 04:02:15.588535 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.588510 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0393ed-openshift-default-55b9654d67-94749"] Oct 30 04:02:15.943481 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:15.943387 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr"] Oct 30 04:02:16.047187 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:16.047099 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:16.109036 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:16.108997 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" path="/var/lib/kubelet/pods/5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b/volumes" Oct 30 04:02:16.148139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:16.148088 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:16.248918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:16.248884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:17.433479 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:17.433449 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:18.010011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:18.009974 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:18.702466 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:18.702434 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:19.595574 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:19.595541 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn"] Oct 30 04:02:19.595821 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:19.595798 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" containerID="cri-o://6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7" gracePeriod=30 Oct 30 04:02:19.602361 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:19.602332 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:19.985557 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:19.985474 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:19.998591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:19.998555 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:20.277367 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:20.277336 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:02:20.944543 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:20.944477 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerName="payload-pre-processing" containerID="cri-o://a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a" gracePeriod=25 Oct 30 04:02:21.031188 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:21.031157 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:21.063780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:21.063728 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:02:21.507250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:21.507212 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:22.495867 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:22.495835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:22.601906 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:22.601876 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:22.711607 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:22.711558 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:23.205664 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:23.205636 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:23.626986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:23.626953 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:24.148200 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.148166 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:24.864147 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:24.864077 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:02:24.876611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.876587 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 04:02:24.959182 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959134 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-credential-socket\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959365 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959222 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-envoy\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959397 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-certs\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959504 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959441 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-kube-api-access-zgf6d\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-kube-api-access-zgf6d\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959504 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959471 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87aded6a-1932-4856-ae97-c18293a8dec6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87aded6a-1932-4856-ae97-c18293a8dec6-istio-podinfo\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959617 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959525 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87aded6a-1932-4856-ae97-c18293a8dec6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87aded6a-1932-4856-ae97-c18293a8dec6-istiod-ca-cert\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959617 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959578 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-istio-token\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-istio-token\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959666 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-socket\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.959841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.959769 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-data\") pod \"87aded6a-1932-4856-ae97-c18293a8dec6\" (UID: \"87aded6a-1932-4856-ae97-c18293a8dec6\") " Oct 30 04:02:24.960614 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.960586 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-data" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:24.961135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.960815 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-credential-socket" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:24.963379 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.962986 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-certs" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:24.963379 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.963184 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87aded6a-1932-4856-ae97-c18293a8dec6-istiod-ca-cert" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:24.963917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.963582 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-envoy" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:24.963917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.963851 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-socket" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:02:24.966194 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.965612 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-istio-token" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:24.966194 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.965852 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-kube-api-access-zgf6d" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "kube-api-access-zgf6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:24.966528 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:24.966507 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87aded6a-1932-4856-ae97-c18293a8dec6-istio-podinfo" pod "87aded6a-1932-4856-ae97-c18293a8dec6" (UID: "87aded6a-1932-4856-ae97-c18293a8dec6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:02:25.061628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061594 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061627 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061642 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061945 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061655 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061945 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061669 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87aded6a-1932-4856-ae97-c18293a8dec6-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061945 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061679 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgf6d\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-kube-api-access-zgf6d\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061945 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061690 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87aded6a-1932-4856-ae97-c18293a8dec6-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061945 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061701 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87aded6a-1932-4856-ae97-c18293a8dec6-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.061945 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.061712 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87aded6a-1932-4856-ae97-c18293a8dec6-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:25.314785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.314068 4442 generic.go:356] "Generic (PLEG): container finished" podID="87aded6a-1932-4856-ae97-c18293a8dec6" containerID="6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7" exitCode=0 Oct 30 04:02:25.314785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.314194 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" event={"ID":"87aded6a-1932-4856-ae97-c18293a8dec6","Type":"ContainerDied","Data":"6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7"} Oct 30 04:02:25.314785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.314245 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" event={"ID":"87aded6a-1932-4856-ae97-c18293a8dec6","Type":"ContainerDied","Data":"7e7534f7ee53ccfa5bbbd15a3e1b79e18ba47bd6d4f2141bf19c0116cc1feddf"} Oct 30 04:02:25.314785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.314273 4442 scope.go:122] "RemoveContainer" containerID="6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7" Oct 30 04:02:25.314785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.314429 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn" Oct 30 04:02:25.330682 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.330660 4442 scope.go:122] "RemoveContainer" containerID="6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7" Oct 30 04:02:25.331046 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:25.331023 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7\": container with ID starting with 6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7 not found: ID does not exist" containerID="6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7" Oct 30 04:02:25.331146 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.331055 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7"} err="failed to get container status \"6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7\": rpc error: code = NotFound desc = could not find container \"6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7\": container with ID starting with 6a9264b4a52e6e6a946763ed5061da7c11b39b555cab3e7916f12104a2890bb7 not found: ID does not exist" Oct 30 04:02:25.365601 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.365565 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn"] Oct 30 04:02:25.375581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.375554 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-372480-openshift-default-85c46d7765-6rktn"] Oct 30 04:02:25.525273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.525237 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:25.605010 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.604912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:25.700876 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:25.700834 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:26.108380 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:26.108346 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" path="/var/lib/kubelet/pods/87aded6a-1932-4856-ae97-c18293a8dec6/volumes" Oct 30 04:02:27.103084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:27.103052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:28.536311 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:28.536280 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:29.413045 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:29.412999 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:30.032939 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.032907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:30.305635 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.305564 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4"] Oct 30 04:02:30.306234 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306218 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" Oct 30 04:02:30.306287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306235 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" Oct 30 04:02:30.306287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306248 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:30.306287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306254 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:30.306287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306269 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" Oct 30 04:02:30.306287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306274 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" Oct 30 04:02:30.306287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306286 4442 cpu_manager.go:414] "containerMap: removing container" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306291 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306302 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306306 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306422 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60232eb-dc73-4290-a115-df93a7b1c6f8" containerName="istio-proxy" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306432 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="87aded6a-1932-4856-ae97-c18293a8dec6" containerName="istio-proxy" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306438 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306446 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e6e45a-fb95-4bd6-9bf5-4362009ff96b" containerName="cleanup" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306453 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b967b29-70ea-4e31-8946-8c4b7ee77558" containerName="cleanup" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306458 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4febf3ae-a71e-4ef8-821e-76fd7b679d66" containerName="cleanup" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306465 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="70550a44-ea69-4758-b024-8f86c4252f3b" containerName="cleanup" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306472 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7673a4d1-cd3f-4ba6-8a12-3a76eac7d16b" containerName="cleanup" Oct 30 04:02:30.306480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.306477 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8b873e-8ca1-43c3-bcc6-aa9834bf39c2" containerName="cleanup" Oct 30 04:02:30.311845 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.311822 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.329563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.329536 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4"] Oct 30 04:02:30.335368 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.335344 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"openshift-service-ca.crt\"" Oct 30 04:02:30.335604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.335462 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"kube-root-ca.crt\"" Oct 30 04:02:30.337254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.337217 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"sub-test-model-547a78-kserve-self-signed-certs\"" Oct 30 04:02:30.337254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.337226 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-547a78\"/\"default-dockercfg-lcxtm\"" Oct 30 04:02:30.341969 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.341941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:30.438451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.438413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-dshm\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.438635 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.438456 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfwfq\" (UniqueName: \"kubernetes.io/projected/f0cc4aba-fc20-4548-b892-941ec5d42570-kube-api-access-nfwfq\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.438730 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.438711 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0cc4aba-fc20-4548-b892-941ec5d42570-tls-certs\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.438851 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.438834 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-home\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.438911 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.438872 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-model-cache\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.438964 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.438942 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-tmp-dir\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.541398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.541352 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0cc4aba-fc20-4548-b892-941ec5d42570-tls-certs\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.541718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.541686 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-home\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.541994 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.541970 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-model-cache\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.542183 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.542155 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-tmp-dir\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.542427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.542408 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-dshm\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.542553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.542538 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfwfq\" (UniqueName: \"kubernetes.io/projected/f0cc4aba-fc20-4548-b892-941ec5d42570-kube-api-access-nfwfq\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.552618 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.552585 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-model-cache\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.570793 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.553911 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-tmp-dir\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.570793 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.555348 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-home\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.593624 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.575691 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0cc4aba-fc20-4548-b892-941ec5d42570-tls-certs\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.594592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.594495 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfwfq\" (UniqueName: \"kubernetes.io/projected/f0cc4aba-fc20-4548-b892-941ec5d42570-kube-api-access-nfwfq\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.604390 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.604361 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-dshm\") pod \"sub-test-model-547a78-kserve-5f5b95d45-dc2s4\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.629549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.629517 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:30.935487 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:30.935463 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4"] Oct 30 04:02:30.936011 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:02:30.935988 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0cc4aba_fc20_4548_b892_941ec5d42570.slice/crio-6d3e7a5b2ce2191bd509dd4a061ae2d8af6a665d4e77b717bff8e341e6a1d35f WatchSource:0}: Error finding container 6d3e7a5b2ce2191bd509dd4a061ae2d8af6a665d4e77b717bff8e341e6a1d35f: Status 404 returned error can't find the container with id 6d3e7a5b2ce2191bd509dd4a061ae2d8af6a665d4e77b717bff8e341e6a1d35f Oct 30 04:02:31.063270 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:31.063235 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:02:31.358307 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:31.358274 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" event={"ID":"f0cc4aba-fc20-4548-b892-941ec5d42570","Type":"ContainerStarted","Data":"fa17ef49969f172efd7809a91dd372ead3539c219d649aa7aea7089a641d57e4"} Oct 30 04:02:31.358489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:31.358315 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" event={"ID":"f0cc4aba-fc20-4548-b892-941ec5d42570","Type":"ContainerStarted","Data":"6d3e7a5b2ce2191bd509dd4a061ae2d8af6a665d4e77b717bff8e341e6a1d35f"} Oct 30 04:02:31.398281 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:31.398222 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" podStartSLOduration=1.398206659 podStartE2EDuration="1.398206659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:31.394345778 +0000 UTC m=+3575.396699014" watchObservedRunningTime="2026-10-30 04:02:31.398206659 +0000 UTC m=+3575.400559856" Oct 30 04:02:31.630444 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:31.630365 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:31.652481 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:31.652443 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:33.081726 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:33.081670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:34.478445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:34.478398 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:35.899127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:35.899082 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:37.378183 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:37.378147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:38.765703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:38.765668 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:40.390109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:40.390078 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:40.630080 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:40.630040 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:40.643736 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:40.643659 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:41.062699 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:02:41.062667 4442 logging.go:55] [core] [Channel #11909 SubChannel #11910]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.216:50052", ServerName: "10.134.0.216:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.216:50052: connect: connection refused" Oct 30 04:02:41.229769 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.229736 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 04:02:41.311804 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.311770 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-kube-api-access-5xcl9\" (UniqueName: \"kubernetes.io/projected/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-kube-api-access-5xcl9\") pod \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " Oct 30 04:02:41.311983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.311897 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-plugins-config-volume\") pod \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\" (UID: \"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5\") " Oct 30 04:02:41.312236 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.312215 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-plugins-config-volume" pod "d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" (UID: "d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:41.312478 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.312463 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:41.313898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.313877 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-kube-api-access-5xcl9" pod "d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" (UID: "d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5"). InnerVolumeSpecName "kube-api-access-5xcl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:41.414211 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.414183 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xcl9\" (UniqueName: \"kubernetes.io/projected/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5-kube-api-access-5xcl9\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:41.431689 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.431661 4442 generic.go:356] "Generic (PLEG): container finished" podID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerID="a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a" exitCode=0 Oct 30 04:02:41.431849 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.431706 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" Oct 30 04:02:41.431849 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.431768 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" event={"ID":"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5","Type":"ContainerDied","Data":"a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a"} Oct 30 04:02:41.431849 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.431828 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" event={"ID":"d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5","Type":"ContainerDied","Data":"eab1b8bfeed341c226ae0359886875aac50a5fb984ce7c2b7ea88975d1cc4087"} Oct 30 04:02:41.432041 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.431858 4442 scope.go:122] "RemoveContainer" containerID="a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a" Oct 30 04:02:41.444257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.444233 4442 scope.go:122] "RemoveContainer" containerID="a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a" Oct 30 04:02:41.447006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.446975 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:02:41.448082 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:41.448056 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a\": container with ID starting with a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a not found: ID does not exist" containerID="a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a" Oct 30 04:02:41.448172 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.448092 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a"} err="failed to get container status \"a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a\": rpc error: code = NotFound desc = could not find container \"a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a\": container with ID starting with a9b93f5f631667a99dadbdc7b361d4b6757867e080d02b4e8b4ca9909cf2389a not found: ID does not exist" Oct 30 04:02:41.481875 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.481806 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr"] Oct 30 04:02:41.485243 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.485224 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr"] Oct 30 04:02:41.889427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:41.889395 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:42.062444 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:42.062405 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b267f8-7c8c7dcc79-kg6gr" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.216:50052\" within 1s: context deadline exceeded" Oct 30 04:02:42.108884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:42.108853 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" path="/var/lib/kubelet/pods/d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5/volumes" Oct 30 04:02:43.446480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:43.446449 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:43.657965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:43.657929 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:02:45.115062 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:45.115029 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:46.806780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:46.806344 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk"] Oct 30 04:02:46.806780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:46.806578 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" podUID="a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" containerName="maas-api" containerID="cri-o://58301e0fad69ada642970eee7baac3900968889279691fac4488f9e4e37b3841" gracePeriod=30 Oct 30 04:02:46.870825 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:46.870732 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:47.591778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.586379 4442 generic.go:356] "Generic (PLEG): container finished" podID="a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" containerID="58301e0fad69ada642970eee7baac3900968889279691fac4488f9e4e37b3841" exitCode=0 Oct 30 04:02:47.591778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.586531 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" event={"ID":"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a","Type":"ContainerDied","Data":"58301e0fad69ada642970eee7baac3900968889279691fac4488f9e4e37b3841"} Oct 30 04:02:47.823921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.823885 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:47.848358 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.848290 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 04:02:47.864158 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.864129 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-maas-api-tls\") pod \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " Oct 30 04:02:47.864158 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.864161 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-kube-api-access-vzg4t\" (UniqueName: \"kubernetes.io/projected/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-kube-api-access-vzg4t\") pod \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " Oct 30 04:02:47.864351 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.864298 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-metrics-certs\") pod \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " Oct 30 04:02:47.864410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.864380 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-openshift-service-ca\") pod \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\" (UID: \"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a\") " Oct 30 04:02:47.864831 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.864802 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-openshift-service-ca" pod "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" (UID: "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:02:47.865008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.864984 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:47.866646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.866624 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-metrics-certs" pod "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" (UID: "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:47.866966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.866934 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-kube-api-access-vzg4t" pod "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" (UID: "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a"). InnerVolumeSpecName "kube-api-access-vzg4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:02:47.866966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.866959 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-maas-api-tls" pod "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" (UID: "a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:02:47.966457 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.966421 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:47.966457 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.966456 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzg4t\" (UniqueName: \"kubernetes.io/projected/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-kube-api-access-vzg4t\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:47.966675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:47.966470 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:02:48.594383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:48.594355 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" Oct 30 04:02:48.594567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:48.594397 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk" event={"ID":"a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a","Type":"ContainerDied","Data":"81e8b933358cd064fa46de4cf639d66c2eee836036c6d83a4f5c8e79545afbbb"} Oct 30 04:02:48.594567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:48.594458 4442 scope.go:122] "RemoveContainer" containerID="58301e0fad69ada642970eee7baac3900968889279691fac4488f9e4e37b3841" Oct 30 04:02:48.655937 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:48.655899 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk"] Oct 30 04:02:48.666643 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:48.666616 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b267f8-76fbd55975-d2gjk"] Oct 30 04:02:49.281058 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:49.281027 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:50.108160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:50.108130 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" path="/var/lib/kubelet/pods/a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a/volumes" Oct 30 04:02:50.766173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:50.766136 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:51.887227 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:51.887181 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:52.204646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:52.204566 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:52.829434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:52.829403 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:53.675391 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:53.675356 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:54.675260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:54.675228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:54.863984 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:02:54.863951 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:02:55.079562 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:55.079531 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:56.233556 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:56.233476 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:56.316798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:56.316744 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:56.613106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:56.613080 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:56.738727 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:56.738699 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:02:56.747366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:56.747341 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:02:56.952222 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:56.952146 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:57.332611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:57.332576 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks"] Oct 30 04:02:57.333115 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:57.332883 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" containerID="cri-o://86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98" gracePeriod=30 Oct 30 04:02:57.355830 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:57.355802 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:58.452741 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:58.452702 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:59.001420 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.001386 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b"] Oct 30 04:02:59.002405 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002376 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" Oct 30 04:02:59.002405 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002402 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" Oct 30 04:02:59.002620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002432 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerName="payload-pre-processing" Oct 30 04:02:59.002620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002440 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerName="payload-pre-processing" Oct 30 04:02:59.002620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002451 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" containerName="maas-api" Oct 30 04:02:59.002620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002457 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" containerName="maas-api" Oct 30 04:02:59.002620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002606 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d38dca31-fcbf-4e8c-babe-a71dfe4fdfc5" containerName="payload-pre-processing" Oct 30 04:02:59.002620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002619 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d210c9-e8b4-4cf0-a5db-a1b0cbf8382a" containerName="maas-api" Oct 30 04:02:59.003409 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.002628 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e37dd5f-96b4-41ae-8e7b-c6ce51cc1a8b" containerName="istio-proxy" Oct 30 04:02:59.008329 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.008306 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.035269 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.035244 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b"] Oct 30 04:02:59.039617 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.039592 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:59.041106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.041000 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"openshift-service-ca.crt\"" Oct 30 04:02:59.041106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.041052 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"kube-root-ca.crt\"" Oct 30 04:02:59.041106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.041076 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"rate-test-model-7d99a4-kserve-self-signed-certs\"" Oct 30 04:02:59.041106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.041078 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-7d99a4\"/\"default-dockercfg-kftdj\"" Oct 30 04:02:59.131658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.131623 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2n7\" (UniqueName: \"kubernetes.io/projected/bd6517be-64ee-4ab8-97e0-26b4d01784f6-kube-api-access-9h2n7\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.131874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.131703 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-dshm\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.131957 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.131877 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tls-certs\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.131957 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.131924 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-model-cache\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.132106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.132054 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-home\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.132106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.132087 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tmp-dir\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.236410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.235476 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tls-certs\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.236410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.235546 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-model-cache\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.236410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.235671 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-home\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.236410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.235698 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tmp-dir\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.236410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.235856 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2n7\" (UniqueName: \"kubernetes.io/projected/bd6517be-64ee-4ab8-97e0-26b4d01784f6-kube-api-access-9h2n7\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.236410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.235893 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-dshm\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.237982 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.237956 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-home\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.238850 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.238827 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tmp-dir\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.239546 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.239191 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-model-cache\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.270234 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.262502 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-dshm\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.270234 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.264847 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2n7\" (UniqueName: \"kubernetes.io/projected/bd6517be-64ee-4ab8-97e0-26b4d01784f6-kube-api-access-9h2n7\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.278374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.278339 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tls-certs\") pod \"rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.335736 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.335697 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:02:59.653960 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.653931 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b"] Oct 30 04:02:59.654339 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:02:59.654308 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd6517be_64ee_4ab8_97e0_26b4d01784f6.slice/crio-3adaf4e8b8b1657e7cd0a276b04b0e6a21670872ac304a93bf0befd6ee21c95a WatchSource:0}: Error finding container 3adaf4e8b8b1657e7cd0a276b04b0e6a21670872ac304a93bf0befd6ee21c95a: Status 404 returned error can't find the container with id 3adaf4e8b8b1657e7cd0a276b04b0e6a21670872ac304a93bf0befd6ee21c95a Oct 30 04:02:59.674290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.674260 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" event={"ID":"bd6517be-64ee-4ab8-97e0-26b4d01784f6","Type":"ContainerStarted","Data":"3adaf4e8b8b1657e7cd0a276b04b0e6a21670872ac304a93bf0befd6ee21c95a"} Oct 30 04:02:59.905460 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:02:59.905374 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:00.682565 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:00.682526 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" event={"ID":"bd6517be-64ee-4ab8-97e0-26b4d01784f6","Type":"ContainerStarted","Data":"1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e"} Oct 30 04:03:00.718652 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:00.718597 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" podStartSLOduration=2.718584715 podStartE2EDuration="2.718584715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:00.717007839 +0000 UTC m=+3604.719361036" watchObservedRunningTime="2026-10-30 04:03:00.718584715 +0000 UTC m=+3604.720937919" Oct 30 04:03:01.291183 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:01.291151 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:01.336563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:01.336522 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:03:02.625533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.625508 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 04:03:02.687582 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687546 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-token\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-token\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.687781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687617 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-data\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.687781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687691 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-credential-socket\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.687781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687712 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6896368e-e630-42b6-94a7-0f588f7c86bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6896368e-e630-42b6-94a7-0f588f7c86bc-istiod-ca-cert\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.687781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687743 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-socket\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.687995 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687825 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-envoy\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.687995 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687898 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-kube-api-access-2jz55\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-kube-api-access-2jz55\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.687995 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.687966 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-podinfo\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.688140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.688004 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-certs\") pod \"6896368e-e630-42b6-94a7-0f588f7c86bc\" (UID: \"6896368e-e630-42b6-94a7-0f588f7c86bc\") " Oct 30 04:03:02.688140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.688033 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-credential-socket" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:02.688575 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.688544 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-socket" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:02.688789 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.688575 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-certs" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:02.688883 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.688813 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-data" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:02.689029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.688989 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.689029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.689013 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.689029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.689028 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.689223 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.689042 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.692557 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.692466 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-kube-api-access-2jz55" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "kube-api-access-2jz55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:02.692557 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.692500 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-token" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:03:02.694903 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.694881 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-podinfo" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:03:02.695040 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.695016 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-envoy" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:03:02.695262 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.695243 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6896368e-e630-42b6-94a7-0f588f7c86bc-istiod-ca-cert" pod "6896368e-e630-42b6-94a7-0f588f7c86bc" (UID: "6896368e-e630-42b6-94a7-0f588f7c86bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:03:02.699332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.699296 4442 generic.go:356] "Generic (PLEG): container finished" podID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerID="86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98" exitCode=0 Oct 30 04:03:02.699431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.699361 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" Oct 30 04:03:02.699493 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.699454 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" event={"ID":"6896368e-e630-42b6-94a7-0f588f7c86bc","Type":"ContainerDied","Data":"86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98"} Oct 30 04:03:02.699556 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.699538 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks" event={"ID":"6896368e-e630-42b6-94a7-0f588f7c86bc","Type":"ContainerDied","Data":"2ada24705f84e1f88a54d77f929e053f2867ae6ff75009bdf33c12a9b4d4b2f1"} Oct 30 04:03:02.699618 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.699570 4442 scope.go:122] "RemoveContainer" containerID="86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98" Oct 30 04:03:02.724398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.724378 4442 scope.go:122] "RemoveContainer" containerID="86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98" Oct 30 04:03:02.724700 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:03:02.724680 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98\": container with ID starting with 86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98 not found: ID does not exist" containerID="86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98" Oct 30 04:03:02.724789 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.724718 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98"} err="failed to get container status \"86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98\": rpc error: code = NotFound desc = could not find container \"86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98\": container with ID starting with 86bb7c6d41f8804012db4c5d793735b14c5594b98ab2bf645ed2c497bbb72c98 not found: ID does not exist" Oct 30 04:03:02.758898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.758868 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks"] Oct 30 04:03:02.764249 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.764226 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b267f8-openshift-default-864488dc79-st6ks"] Oct 30 04:03:02.786157 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.786129 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:02.791222 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.791197 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.791354 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.791225 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6896368e-e630-42b6-94a7-0f588f7c86bc-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.791354 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.791239 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.791354 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.791252 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jz55\" (UniqueName: \"kubernetes.io/projected/6896368e-e630-42b6-94a7-0f588f7c86bc-kube-api-access-2jz55\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:02.791354 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:02.791265 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6896368e-e630-42b6-94a7-0f588f7c86bc-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:03:04.109436 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:04.109399 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" path="/var/lib/kubelet/pods/6896368e-e630-42b6-94a7-0f588f7c86bc/volumes" Oct 30 04:03:04.297394 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:04.297367 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:05.894497 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:05.894466 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:07.507320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:07.507274 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:08.894894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:08.894854 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:09.335978 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:09.335947 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:03:09.348138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:09.348106 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:03:09.756456 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:09.756433 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:03:10.405553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:10.405498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:11.843233 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:11.843195 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:13.265301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:13.265266 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:14.648993 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:14.648955 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:16.253746 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:16.253714 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:17.918315 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:17.918282 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:19.274236 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:19.274200 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:20.715555 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:20.715520 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:21.432949 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:21.432908 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:03:22.204177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:22.204142 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:23.797345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:23.797309 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:24.861605 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:03:24.861570 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:03:25.202963 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:25.202876 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:26.090034 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.089996 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw"] Oct 30 04:03:26.090701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.090685 4442 cpu_manager.go:414] "containerMap: removing container" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" Oct 30 04:03:26.090802 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.090703 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" Oct 30 04:03:26.090851 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.090828 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="6896368e-e630-42b6-94a7-0f588f7c86bc" containerName="istio-proxy" Oct 30 04:03:26.096061 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.096030 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.118957 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.118925 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"default-dockercfg-tvn88\"" Oct 30 04:03:26.121354 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.121317 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw"] Oct 30 04:03:26.123530 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.123506 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"kube-root-ca.crt\"" Oct 30 04:03:26.123530 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.123506 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"openshift-service-ca.crt\"" Oct 30 04:03:26.123689 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.123515 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-852107\"/\"rate-test-model-852107-kserve-self-signed-certs\"" Oct 30 04:03:26.131686 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.131667 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:26.202834 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.202801 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-home\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.203006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.202851 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpk8\" (UniqueName: \"kubernetes.io/projected/452ef5f7-56c5-4b2b-863c-58414354fca5-kube-api-access-4fpk8\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.203006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.202958 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-dshm\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.203006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.202976 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-tmp-dir\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.203138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.203009 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-model-cache\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.203175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.203141 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ef5f7-56c5-4b2b-863c-58414354fca5-tls-certs\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.304565 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.304521 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-home\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.304791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.304579 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpk8\" (UniqueName: \"kubernetes.io/projected/452ef5f7-56c5-4b2b-863c-58414354fca5-kube-api-access-4fpk8\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.304791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.304656 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-dshm\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.304791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.304689 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-tmp-dir\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.304791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.304728 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-model-cache\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.305023 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.304849 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ef5f7-56c5-4b2b-863c-58414354fca5-tls-certs\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.305071 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.305021 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-home\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.305177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.305153 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-tmp-dir\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.305177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.305167 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-model-cache\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.307009 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.306984 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-dshm\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.307389 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.307370 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ef5f7-56c5-4b2b-863c-58414354fca5-tls-certs\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.359063 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.358983 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpk8\" (UniqueName: \"kubernetes.io/projected/452ef5f7-56c5-4b2b-863c-58414354fca5-kube-api-access-4fpk8\") pod \"rate-test-model-852107-kserve-7c896c87d9-mwzlw\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.419025 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.418989 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:26.697388 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.697299 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:26.729605 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.729574 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw"] Oct 30 04:03:26.731032 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:03:26.731009 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod452ef5f7_56c5_4b2b_863c_58414354fca5.slice/crio-6e736e2d0dc4c05d5e23a59525470621ad07ba0ff28a97e9a867e928ac28817d WatchSource:0}: Error finding container 6e736e2d0dc4c05d5e23a59525470621ad07ba0ff28a97e9a867e928ac28817d: Status 404 returned error can't find the container with id 6e736e2d0dc4c05d5e23a59525470621ad07ba0ff28a97e9a867e928ac28817d Oct 30 04:03:26.908723 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.908685 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" event={"ID":"452ef5f7-56c5-4b2b-863c-58414354fca5","Type":"ContainerStarted","Data":"e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e"} Oct 30 04:03:26.908949 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.908736 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" event={"ID":"452ef5f7-56c5-4b2b-863c-58414354fca5","Type":"ContainerStarted","Data":"6e736e2d0dc4c05d5e23a59525470621ad07ba0ff28a97e9a867e928ac28817d"} Oct 30 04:03:26.946225 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:26.946151 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" podStartSLOduration=0.946138354 podStartE2EDuration="946.138354ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:26.943786275 +0000 UTC m=+3630.946139494" watchObservedRunningTime="2026-10-30 04:03:26.946138354 +0000 UTC m=+3630.948491550" Oct 30 04:03:27.420061 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:27.420020 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:28.199025 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:28.198996 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:29.608080 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:29.608049 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:31.085647 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:31.085604 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:32.526787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:32.526738 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:33.959440 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:33.959405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:35.403046 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:35.403007 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:36.419954 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:36.419919 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:36.432994 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:36.432956 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:36.985625 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:36.985595 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:03:37.089122 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:37.089091 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:38.485771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:38.485730 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:40.002859 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:40.002772 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:41.502462 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:41.502428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:43.184562 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:43.184528 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:44.581852 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:44.581803 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:46.108930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:46.108897 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:47.510844 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:47.510797 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:48.982456 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:48.982409 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:50.481447 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:50.481412 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:51.908768 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:51.908730 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:52.009745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:52.009703 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:52.944944 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:52.944910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:53.391722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:53.391677 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:54.029414 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:54.029384 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:54.804573 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:54.804540 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:54.861373 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:03:54.861337 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:03:54.911104 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:54.911075 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:56.396160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:56.396126 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:57.674396 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:57.674364 4442 scope.go:122] "RemoveContainer" containerID="78bad4c9ca9e980a8a6ea23d3d8bad059db1701d5de101f54df17754620d73aa" Oct 30 04:03:57.806421 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:57.806388 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:59.279896 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:59.279855 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:59.923132 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:03:59.923097 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:00.019933 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:00.019898 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:00.884540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:00.884489 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:02.308248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:02.308216 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:03.775975 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:03.775944 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:05.170234 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:05.170199 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:06.655564 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:06.655536 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:06.752272 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:06.752235 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:07.995457 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:07.995425 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:08.093961 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:08.093930 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:08.690666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:08.690621 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:09.515490 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:09.515454 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:04:10.207022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:10.206990 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:11.841904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:11.841871 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:13.284123 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:13.284084 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:13.376311 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:13.376275 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:13.490278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:13.490228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:13.859140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:13.859113 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:14.263629 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:14.263583 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:15.013271 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.013239 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:15.103073 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.103037 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:15.519062 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.519025 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj"] Oct 30 04:04:15.523338 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.523315 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.543665 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.543645 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-30e3dd-openshift-default-dockercfg-dx57w\"" Oct 30 04:04:15.564889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.564865 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj"] Oct 30 04:04:15.610783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.610715 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:15.661039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.660999 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-token\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.661427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.661399 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-data\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.661628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.661606 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-credential-socket\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.661807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.661789 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzrg\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-kube-api-access-bnzrg\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.661999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.661980 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-podinfo\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.662157 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.662139 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-envoy\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.662282 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.662268 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6560d-6e76-494a-86fd-74216b73ccc1-istiod-ca-cert\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.662412 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.662398 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-certs\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.662714 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.662691 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-socket\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.740619 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.740446 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:15.772874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.772598 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-podinfo\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.772874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.772681 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-envoy\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.772874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.772712 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6560d-6e76-494a-86fd-74216b73ccc1-istiod-ca-cert\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.772874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.772771 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-certs\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.773299 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.772925 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-socket\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.773299 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.773042 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-token\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.773299 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.773075 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-data\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.773299 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.773127 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-credential-socket\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.773299 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.773174 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzrg\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-kube-api-access-bnzrg\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.779519 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.774288 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-credential-socket\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.779519 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.774298 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-certs\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.779519 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.774534 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-socket\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.779519 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.775254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-data\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.779519 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.778139 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6560d-6e76-494a-86fd-74216b73ccc1-istiod-ca-cert\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.780826 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.780797 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-envoy\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.781395 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.781372 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-podinfo\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.798125 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.798093 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-token\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.799121 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.799069 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzrg\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-kube-api-access-bnzrg\") pod \"e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:15.839967 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:15.839932 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:16.169102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.169079 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj"] Oct 30 04:04:16.169885 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:04:16.169851 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a6560d_6e76_494a_86fd_74216b73ccc1.slice/crio-2a119d08206634c85a4db38221457627dbfafe6245496c0a09da386176b3b438 WatchSource:0}: Error finding container 2a119d08206634c85a4db38221457627dbfafe6245496c0a09da386176b3b438: Status 404 returned error can't find the container with id 2a119d08206634c85a4db38221457627dbfafe6245496c0a09da386176b3b438 Oct 30 04:04:16.172547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.172516 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:04:16.172644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.172629 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:04:16.172683 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.172663 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:04:16.290986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.290935 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" event={"ID":"c5a6560d-6e76-494a-86fd-74216b73ccc1","Type":"ContainerStarted","Data":"4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a"} Oct 30 04:04:16.291181 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.291003 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" event={"ID":"c5a6560d-6e76-494a-86fd-74216b73ccc1","Type":"ContainerStarted","Data":"2a119d08206634c85a4db38221457627dbfafe6245496c0a09da386176b3b438"} Oct 30 04:04:16.333114 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.333058 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podStartSLOduration=1.33304134 podStartE2EDuration="1.33304134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:16.32980865 +0000 UTC m=+3680.332161846" watchObservedRunningTime="2026-10-30 04:04:16.33304134 +0000 UTC m=+3680.335394537" Oct 30 04:04:16.675876 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.675829 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:16.840147 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:16.840114 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:17.846595 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:17.846552 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:18.200220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:18.200126 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:18.320907 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:18.320867 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:18.321987 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:18.321963 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:04:18.734362 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:18.734329 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:18.832151 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:18.832115 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:19.716297 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:19.716266 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:20.457023 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.456994 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:20.555963 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.555926 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:20.648027 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.647996 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6"] Oct 30 04:04:20.652531 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.652509 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:20.668927 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.668895 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6"] Oct 30 04:04:20.674942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.674908 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-30e3dd\"" Oct 30 04:04:20.734247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.729421 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-30e3dd-dockercfg-qtgjh\"" Oct 30 04:04:20.743695 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.741864 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13125b72-bd40-4ec9-b225-54b81ceb78a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:20.744384 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.744130 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzlvl\" (UniqueName: \"kubernetes.io/projected/13125b72-bd40-4ec9-b225-54b81ceb78a1-kube-api-access-lzlvl\") pod \"payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:20.750445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.750417 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44"] Oct 30 04:04:20.810372 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.810340 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:20.846231 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.846197 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13125b72-bd40-4ec9-b225-54b81ceb78a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:20.847406 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.847380 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzlvl\" (UniqueName: \"kubernetes.io/projected/13125b72-bd40-4ec9-b225-54b81ceb78a1-kube-api-access-lzlvl\") pod \"payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:20.848051 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.847281 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13125b72-bd40-4ec9-b225-54b81ceb78a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:20.871133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.871101 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44"] Oct 30 04:04:20.872075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.872047 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzlvl\" (UniqueName: \"kubernetes.io/projected/13125b72-bd40-4ec9-b225-54b81ceb78a1-kube-api-access-lzlvl\") pod \"payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:20.949698 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.949663 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:20.949942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.949903 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wkn\" (UniqueName: \"kubernetes.io/projected/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-kube-api-access-p5wkn\") pod \"payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:20.970357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:20.970328 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:21.053358 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.052453 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:21.055360 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.053596 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wkn\" (UniqueName: \"kubernetes.io/projected/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-kube-api-access-p5wkn\") pod \"payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:21.055360 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.053447 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:21.071656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.071632 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wkn\" (UniqueName: \"kubernetes.io/projected/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-kube-api-access-p5wkn\") pod \"payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:21.145682 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.145654 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:21.285876 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:04:21.283058 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13125b72_bd40_4ec9_b225_54b81ceb78a1.slice/crio-b8d6b87bc80ed84f0982ba747c60d63dc81bd7b2f4aab3568bdc358bdb4c6da8 WatchSource:0}: Error finding container b8d6b87bc80ed84f0982ba747c60d63dc81bd7b2f4aab3568bdc358bdb4c6da8: Status 404 returned error can't find the container with id b8d6b87bc80ed84f0982ba747c60d63dc81bd7b2f4aab3568bdc358bdb4c6da8 Oct 30 04:04:21.295309 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.295276 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6"] Oct 30 04:04:21.339659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.339630 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" event={"ID":"13125b72-bd40-4ec9-b225-54b81ceb78a1","Type":"ContainerStarted","Data":"b8d6b87bc80ed84f0982ba747c60d63dc81bd7b2f4aab3568bdc358bdb4c6da8"} Oct 30 04:04:21.457932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.457903 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44"] Oct 30 04:04:21.458459 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:04:21.458438 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fce5766_abf0_49e6_9f3f_09c2a86f0b81.slice/crio-b2a6e3e5d5179e2a25597e0c77967db3b3e7a76b6d8591b97353fc3245423391 WatchSource:0}: Error finding container b2a6e3e5d5179e2a25597e0c77967db3b3e7a76b6d8591b97353fc3245423391: Status 404 returned error can't find the container with id b2a6e3e5d5179e2a25597e0c77967db3b3e7a76b6d8591b97353fc3245423391 Oct 30 04:04:21.481510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:21.481480 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:22.348189 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:22.348141 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" event={"ID":"13125b72-bd40-4ec9-b225-54b81ceb78a1","Type":"ContainerStarted","Data":"ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427"} Oct 30 04:04:22.348720 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:22.348243 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:22.349938 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:22.349910 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" event={"ID":"8fce5766-abf0-49e6-9f3f-09c2a86f0b81","Type":"ContainerStarted","Data":"5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04"} Oct 30 04:04:22.350068 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:22.349959 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" event={"ID":"8fce5766-abf0-49e6-9f3f-09c2a86f0b81","Type":"ContainerStarted","Data":"b2a6e3e5d5179e2a25597e0c77967db3b3e7a76b6d8591b97353fc3245423391"} Oct 30 04:04:22.350068 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:22.350014 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:22.386624 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:22.386573 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" podStartSLOduration=2.386558899 podStartE2EDuration="2.386558899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:22.383513324 +0000 UTC m=+3686.385866521" watchObservedRunningTime="2026-10-30 04:04:22.386558899 +0000 UTC m=+3686.388912106" Oct 30 04:04:22.421695 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:22.421642 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" podStartSLOduration=2.421628756 podStartE2EDuration="2.421628756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:22.418056586 +0000 UTC m=+3686.420409785" watchObservedRunningTime="2026-10-30 04:04:22.421628756 +0000 UTC m=+3686.423981966" Oct 30 04:04:23.063735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:23.063681 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:24.550356 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:24.550321 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:24.860879 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:04:24.860798 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:04:25.998859 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:25.998825 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:27.510124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:27.510090 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:28.965685 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:28.965650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:30.354209 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:30.354176 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:31.807402 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:31.807370 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:33.275040 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:33.275008 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:33.373410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:33.372929 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:04:33.374963 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:33.374937 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:04:34.873176 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:34.873134 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:36.368599 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:36.368560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:37.985255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:37.985217 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:39.365842 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:39.365810 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:40.895956 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:40.895919 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:41.880963 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:41.880915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:42.391071 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:42.391034 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:43.783492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:43.783454 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:45.179077 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:45.179041 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:46.595015 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:46.594981 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:46.986817 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:46.986725 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln"] Oct 30 04:04:47.025155 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:47.025109 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s"] Oct 30 04:04:47.124908 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:47.124868 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:47.224893 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:47.224594 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:48.299523 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:48.299487 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:48.404150 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:48.404118 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:49.783002 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:49.782964 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:51.208344 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:51.208309 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:51.987618 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:51.987582 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" podUID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" containerName="payload-processing" containerID="cri-o://a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65" gracePeriod=25 Oct 30 04:04:52.026439 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:52.026404 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" podUID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" containerName="payload-pre-processing" containerID="cri-o://3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553" gracePeriod=25 Oct 30 04:04:52.203193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:52.203157 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:52.703017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:52.702985 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:54.160989 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:54.160943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:54.237595 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:54.237559 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:54.340994 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:54.340022 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:54.582098 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:54.582067 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:54.867415 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:04:54.867332 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:04:54.942910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:54.942864 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:55.490924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:55.490881 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:55.681199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:55.681142 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:57.296389 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.296352 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8"] Oct 30 04:04:57.301366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.301345 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.319721 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.319695 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8"] Oct 30 04:04:57.323515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.323490 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:57.325605 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.325579 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-1a1b45-openshift-default-dockercfg-gx88g\"" Oct 30 04:04:57.354740 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.354693 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723efe3e-62f7-48ba-b1ed-c15032afd156-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.354935 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.354794 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.354935 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.354890 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.355065 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.354953 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9lq\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-kube-api-access-nc9lq\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.355065 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.355008 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.355065 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.355032 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.355250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.355088 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.355250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.355162 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.355250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.355230 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456197 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456165 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456387 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456231 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456387 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456326 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723efe3e-62f7-48ba-b1ed-c15032afd156-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456387 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456349 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456397 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456426 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9lq\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-kube-api-access-nc9lq\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456473 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456508 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456548 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456952 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456616 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-data\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.456952 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.456821 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.457084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.457052 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-credential-socket\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.457084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.457075 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-certs\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.457210 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.457150 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723efe3e-62f7-48ba-b1ed-c15032afd156-istiod-ca-cert\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.458907 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.458889 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-envoy\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.459250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.459231 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-podinfo\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.474481 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.474449 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-token\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.474601 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.474514 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9lq\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-kube-api-access-nc9lq\") pod \"e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.618726 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.618643 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:57.945814 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:04:57.945780 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723efe3e_62f7_48ba_b1ed_c15032afd156.slice/crio-94b44d6173a387d59d7ccc3d2cac197764170e30d32a7c886b7676905c9552a3 WatchSource:0}: Error finding container 94b44d6173a387d59d7ccc3d2cac197764170e30d32a7c886b7676905c9552a3: Status 404 returned error can't find the container with id 94b44d6173a387d59d7ccc3d2cac197764170e30d32a7c886b7676905c9552a3 Oct 30 04:04:57.946034 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.946009 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8"] Oct 30 04:04:57.949136 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.949102 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:04:57.949255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.949168 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:04:57.949255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:57.949220 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:04:58.654373 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:58.654330 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" event={"ID":"723efe3e-62f7-48ba-b1ed-c15032afd156","Type":"ContainerStarted","Data":"75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2"} Oct 30 04:04:58.654805 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:58.654431 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" event={"ID":"723efe3e-62f7-48ba-b1ed-c15032afd156","Type":"ContainerStarted","Data":"94b44d6173a387d59d7ccc3d2cac197764170e30d32a7c886b7676905c9552a3"} Oct 30 04:04:58.703813 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:58.703741 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podStartSLOduration=1.7037278580000002 podStartE2EDuration="1.703727858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:58.703001302 +0000 UTC m=+3722.705354500" watchObservedRunningTime="2026-10-30 04:04:58.703727858 +0000 UTC m=+3722.706081054" Oct 30 04:04:58.792833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:58.792777 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:59.150232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:59.150197 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" podUID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:04:59.618877 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:59.618831 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:04:59.620895 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:59.620866 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 04:04:59.621063 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:04:59.620923 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 04:05:00.155990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:00.155948 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" podUID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:05:00.414993 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:00.414909 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:00.619075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:00.619038 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 04:05:00.619265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:00.619096 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 04:05:01.620018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:01.619984 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 04:05:01.620522 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:01.620044 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 04:05:02.060063 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:02.060032 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:02.619112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:02.619079 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 04:05:02.619308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:02.619133 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 04:05:03.619251 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:03.619212 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 04:05:03.619696 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:03.619282 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 04:05:03.631338 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:03.631300 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:04.619392 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:04.619352 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 04:05:04.619829 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:04.619415 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 04:05:05.003606 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:05.003571 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:05.625041 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:05.625010 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:05:05.625464 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:05.625409 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:05:05.626191 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:05.626170 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:05:05.640230 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:05.640208 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:06.321827 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:06.321780 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:05:07.224650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:07.224620 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:09.107847 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:09.107807 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:09.150620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:09.150540 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" podUID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:05:09.628152 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:09.628111 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:09.732284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:09.732211 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:10.156131 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:10.156098 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" podUID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:05:10.319806 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:10.319775 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:10.736127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:10.736095 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv"] Oct 30 04:05:10.736514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:10.736484 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" containerID="cri-o://0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62" gracePeriod=30 Oct 30 04:05:10.766785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:10.766720 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:10.919783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:10.918366 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:11.580630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:11.580597 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:12.358552 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.358525 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 04:05:12.423169 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.423142 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 04:05:12.475968 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.475937 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-plugins-config-volume\") pod \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " Oct 30 04:05:12.476152 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.476056 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-kube-api-access-jb2df\" (UniqueName: \"kubernetes.io/projected/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-kube-api-access-jb2df\") pod \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\" (UID: \"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e\") " Oct 30 04:05:12.476445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.476420 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-plugins-config-volume" pod "8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" (UID: "8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:12.476824 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.476796 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:12.478823 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.478733 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-kube-api-access-jb2df" pod "8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" (UID: "8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e"). InnerVolumeSpecName "kube-api-access-jb2df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:12.578124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.578045 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-plugins-config-volume\") pod \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " Oct 30 04:05:12.578277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.578178 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-kube-api-access-wk9ct\" (UniqueName: \"kubernetes.io/projected/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-kube-api-access-wk9ct\") pod \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\" (UID: \"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78\") " Oct 30 04:05:12.578496 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.578475 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-plugins-config-volume" pod "ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" (UID: "ac5d678d-d1c0-4f80-bbb6-2ee5af311d78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:12.578705 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.578692 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:12.578743 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.578709 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb2df\" (UniqueName: \"kubernetes.io/projected/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e-kube-api-access-jb2df\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:12.580457 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.580426 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-kube-api-access-wk9ct" pod "ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" (UID: "ac5d678d-d1c0-4f80-bbb6-2ee5af311d78"). InnerVolumeSpecName "kube-api-access-wk9ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:12.619593 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.619556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:12.701777 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.701263 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9ct\" (UniqueName: \"kubernetes.io/projected/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78-kube-api-access-wk9ct\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:12.820440 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.820405 4442 generic.go:356] "Generic (PLEG): container finished" podID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" containerID="a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65" exitCode=0 Oct 30 04:05:12.820643 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.820579 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" Oct 30 04:05:12.821146 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.821121 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" event={"ID":"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e","Type":"ContainerDied","Data":"a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65"} Oct 30 04:05:12.821319 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.821178 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln" event={"ID":"8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e","Type":"ContainerDied","Data":"f810fa7d65d55c5223541d8ab04bddf90bd3877f57b7fe4494f5da597e24fad7"} Oct 30 04:05:12.821319 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.821212 4442 scope.go:122] "RemoveContainer" containerID="a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65" Oct 30 04:05:12.823976 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.823923 4442 generic.go:356] "Generic (PLEG): container finished" podID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" containerID="3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553" exitCode=0 Oct 30 04:05:12.824254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.824235 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" event={"ID":"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78","Type":"ContainerDied","Data":"3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553"} Oct 30 04:05:12.824356 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.824282 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" event={"ID":"ac5d678d-d1c0-4f80-bbb6-2ee5af311d78","Type":"ContainerDied","Data":"5b697b39787a5fa1a8a3a172a6786a97309537cd4ec4c9bce8595ec47ab44a0d"} Oct 30 04:05:12.824429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.824360 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s" Oct 30 04:05:12.835770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.835687 4442 scope.go:122] "RemoveContainer" containerID="a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65" Oct 30 04:05:12.836056 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:05:12.836032 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65\": container with ID starting with a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65 not found: ID does not exist" containerID="a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65" Oct 30 04:05:12.836134 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.836065 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65"} err="failed to get container status \"a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65\": rpc error: code = NotFound desc = could not find container \"a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65\": container with ID starting with a1eea7e79bc45f617c2a86ad9eedc2020d81dcb38faac3d3139fee5e69ed1b65 not found: ID does not exist" Oct 30 04:05:12.836134 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.836088 4442 scope.go:122] "RemoveContainer" containerID="3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553" Oct 30 04:05:12.847050 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.847033 4442 scope.go:122] "RemoveContainer" containerID="3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553" Oct 30 04:05:12.847345 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:05:12.847327 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553\": container with ID starting with 3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553 not found: ID does not exist" containerID="3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553" Oct 30 04:05:12.847398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.847351 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553"} err="failed to get container status \"3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553\": rpc error: code = NotFound desc = could not find container \"3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553\": container with ID starting with 3d4ba090462d16626d121ff972f13af669113cdc05a2c25ef4f7117366ea3553 not found: ID does not exist" Oct 30 04:05:12.869991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.869956 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln"] Oct 30 04:05:12.873182 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.873157 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-441461-5bd69b7b5b-8cfln"] Oct 30 04:05:12.933467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.933427 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s"] Oct 30 04:05:12.936610 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:12.936588 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-441461-54857d6695-8gc5s"] Oct 30 04:05:14.109594 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:14.109560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:14.112217 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:14.112187 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" path="/var/lib/kubelet/pods/8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e/volumes" Oct 30 04:05:14.112898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:14.112877 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" path="/var/lib/kubelet/pods/ac5d678d-d1c0-4f80-bbb6-2ee5af311d78/volumes" Oct 30 04:05:15.525325 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:15.525290 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:16.034175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.034146 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 04:05:16.052446 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052421 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-podinfo\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052502 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istiod-ca-cert\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052547 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-data\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052579 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-credential-socket\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052667 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-token\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-token\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052719 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-socket\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052824 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-kube-api-access-jfnkl\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-kube-api-access-jfnkl\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052874 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-certs\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.052925 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-envoy\") pod \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\" (UID: \"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f\") " Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053013 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-data" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053091 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-credential-socket" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053201 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-certs" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053365 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istiod-ca-cert" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:05:16.053659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053628 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-socket" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:16.054735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053675 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.054735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053698 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.054735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053713 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.054735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.053726 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.055031 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.054955 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-podinfo" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:05:16.055639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.055613 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-kube-api-access-jfnkl" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "kube-api-access-jfnkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:16.055990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.055966 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-envoy" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:16.056081 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.056039 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-token" pod "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" (UID: "388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:16.154886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.154854 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfnkl\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-kube-api-access-jfnkl\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.154886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.154885 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.155095 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.154898 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.155095 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.154910 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.155095 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.154923 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:16.855628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.855584 4442 generic.go:356] "Generic (PLEG): container finished" podID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerID="0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62" exitCode=0 Oct 30 04:05:16.856138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.855718 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" Oct 30 04:05:16.856138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.855716 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" event={"ID":"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f","Type":"ContainerDied","Data":"0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62"} Oct 30 04:05:16.856138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.855854 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv" event={"ID":"388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f","Type":"ContainerDied","Data":"ecd2f8143786ddf6883d541172c88fdefa1d5af90ce412256533638ed2191028"} Oct 30 04:05:16.856138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.855877 4442 scope.go:122] "RemoveContainer" containerID="0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62" Oct 30 04:05:16.866807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.866582 4442 scope.go:122] "RemoveContainer" containerID="17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c" Oct 30 04:05:16.876918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.876900 4442 scope.go:122] "RemoveContainer" containerID="0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62" Oct 30 04:05:16.877236 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:05:16.877216 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62\": container with ID starting with 0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62 not found: ID does not exist" containerID="0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62" Oct 30 04:05:16.877301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.877247 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62"} err="failed to get container status \"0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62\": rpc error: code = NotFound desc = could not find container \"0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62\": container with ID starting with 0e03af8815c495878642edeea7db49ae356fcb752d867cbcb98e1c553de74f62 not found: ID does not exist" Oct 30 04:05:16.877301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.877263 4442 scope.go:122] "RemoveContainer" containerID="17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c" Oct 30 04:05:16.877509 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:05:16.877494 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c\": container with ID starting with 17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c not found: ID does not exist" containerID="17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c" Oct 30 04:05:16.877568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.877512 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c"} err="failed to get container status \"17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c\": rpc error: code = NotFound desc = could not find container \"17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c\": container with ID starting with 17bd8be2322dc1afd3cdee6d9c3414f90276b262264e7943bd260acbe6fda95c not found: ID does not exist" Oct 30 04:05:16.918816 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.918780 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:16.919540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.919513 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv"] Oct 30 04:05:16.928277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:16.928248 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-441461-openshift-default-7cdc576fc7-7lszv"] Oct 30 04:05:17.865180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:17.865147 4442 generic.go:356] "Generic (PLEG): container finished" podID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerID="4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a" exitCode=0 Oct 30 04:05:17.865655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:17.865216 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" event={"ID":"c5a6560d-6e76-494a-86fd-74216b73ccc1","Type":"ContainerDied","Data":"4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a"} Oct 30 04:05:17.865742 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:17.865649 4442 scope.go:122] "RemoveContainer" containerID="4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a" Oct 30 04:05:17.867436 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:17.867407 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:17.867506 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:17.867481 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:17.867545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:17.867512 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:17.939998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:17.939966 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:18.038482 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:18.038432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:18.115357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:18.115322 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" path="/var/lib/kubelet/pods/388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f/volumes" Oct 30 04:05:18.321234 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:18.321193 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:05:18.671151 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:18.671053 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:18.875797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:18.875736 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" event={"ID":"c5a6560d-6e76-494a-86fd-74216b73ccc1","Type":"ContainerStarted","Data":"e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a"} Oct 30 04:05:19.840712 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:19.840679 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:05:19.841662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:19.841641 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 04:05:19.841724 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:19.841688 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 04:05:20.324009 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:20.323929 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:20.813379 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:20.813317 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:20.842269 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:20.842234 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 04:05:20.842471 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:20.842314 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 04:05:21.841300 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:21.841263 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 04:05:21.841744 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:21.841317 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 04:05:21.963152 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:21.963116 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:22.840773 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:22.840729 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 04:05:22.841083 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:22.840817 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 04:05:23.600781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:23.586293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:23.699598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:23.699560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:23.841314 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:23.841281 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 04:05:23.841498 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:23.841337 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 04:05:24.846022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:24.845994 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:05:24.846507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:24.846482 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:05:24.847313 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:24.847293 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:05:24.861996 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:05:24.861967 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:05:25.207194 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:25.207102 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:26.694718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:26.694686 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:27.296378 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.296345 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs"] Oct 30 04:05:27.297116 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297093 4442 cpu_manager.go:414] "containerMap: removing container" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" Oct 30 04:05:27.297116 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297113 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" Oct 30 04:05:27.297305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297146 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" containerName="payload-pre-processing" Oct 30 04:05:27.297305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297154 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" containerName="payload-pre-processing" Oct 30 04:05:27.297305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297166 4442 cpu_manager.go:414] "containerMap: removing container" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" Oct 30 04:05:27.297305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297171 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" Oct 30 04:05:27.297305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297179 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" containerName="payload-processing" Oct 30 04:05:27.297305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297184 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" containerName="payload-processing" Oct 30 04:05:27.297516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297320 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5d678d-d1c0-4f80-bbb6-2ee5af311d78" containerName="payload-pre-processing" Oct 30 04:05:27.297516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297332 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" Oct 30 04:05:27.297516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.297341 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bcd1f09-7dbb-4255-b670-a6f00f7d9a1e" containerName="payload-processing" Oct 30 04:05:27.302284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.302260 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.316011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.315992 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-da4553-openshift-default-dockercfg-2mt2c\"" Oct 30 04:05:27.331212 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.331179 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs"] Oct 30 04:05:27.484874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.484840 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-certs\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.484894 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istiod-ca-cert\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.484960 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.484991 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-data\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.485010 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-token\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.485071 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-credential-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.485218 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-envoy\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.485272 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-podinfo\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.485455 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.485318 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr69d\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-kube-api-access-jr69d\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.586939 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.586842 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-certs\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.586939 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.586927 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istiod-ca-cert\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.586956 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.586985 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-data\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587010 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-token\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587051 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-credential-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587103 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-envoy\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587140 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-podinfo\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587181 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr69d\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-kube-api-access-jr69d\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587517 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587671 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587637 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-data\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587739 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587679 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-credential-socket\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.587859 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.587840 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-certs\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.588194 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.588167 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istiod-ca-cert\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.589716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.589689 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-envoy\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.590260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.590221 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-podinfo\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.605365 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.605342 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-token\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.605538 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.605519 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr69d\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-kube-api-access-jr69d\") pod \"e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.616463 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.616440 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:27.931393 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.931360 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs"] Oct 30 04:05:27.931393 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:05:27.931382 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode59a5ab2_179c_47fc_a7c3_4dae630f458d.slice/crio-45e74ea7925f224291fbcd85753d6f428fd5aa13af9ee0662dd2e93301ff3459 WatchSource:0}: Error finding container 45e74ea7925f224291fbcd85753d6f428fd5aa13af9ee0662dd2e93301ff3459: Status 404 returned error can't find the container with id 45e74ea7925f224291fbcd85753d6f428fd5aa13af9ee0662dd2e93301ff3459 Oct 30 04:05:27.934341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.934310 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:27.934429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.934377 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:27.934429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:27.934405 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:28.002390 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:28.002361 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" event={"ID":"e59a5ab2-179c-47fc-a7c3-4dae630f458d","Type":"ContainerStarted","Data":"45e74ea7925f224291fbcd85753d6f428fd5aa13af9ee0662dd2e93301ff3459"} Oct 30 04:05:28.269294 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:28.269239 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:29.021293 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:29.021250 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" event={"ID":"e59a5ab2-179c-47fc-a7c3-4dae630f458d","Type":"ContainerStarted","Data":"b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db"} Oct 30 04:05:29.104782 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:29.096102 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" podStartSLOduration=2.096079045 podStartE2EDuration="2.096079045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:29.072327455 +0000 UTC m=+3753.074680674" watchObservedRunningTime="2026-10-30 04:05:29.096079045 +0000 UTC m=+3753.098432254" Oct 30 04:05:29.617613 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:29.617576 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:29.619672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:29.619631 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 04:05:29.619852 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:29.619697 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 04:05:29.858982 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:29.858948 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:30.617524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:30.617485 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 04:05:30.617990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:30.617540 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 04:05:31.358805 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:31.358737 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:31.617853 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:31.617730 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 04:05:31.617853 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:31.617836 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 04:05:32.617644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:32.617604 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 04:05:32.617878 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:32.617671 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 04:05:33.031614 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:33.031574 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:33.617716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:33.617683 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 04:05:33.617986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:33.617769 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 04:05:34.150997 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:34.150959 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:34.538073 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:34.538023 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9"] Oct 30 04:05:34.563484 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:34.563408 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:34.637366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:34.637333 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:34.643634 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:34.640609 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:34.644178 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:34.644156 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:05:34.702619 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:34.702590 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:35.079942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:35.079906 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:35.652222 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:35.652190 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:36.230291 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:36.230258 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:36.327446 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:36.327403 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:36.766463 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:36.766429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:37.036505 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:37.036415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:37.987596 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:37.987548 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:39.487632 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:39.487603 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:40.885956 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:40.885919 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:40.924347 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:40.924314 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:05:40.986006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:40.985978 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:41.088334 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:41.088300 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:41.212826 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:41.205398 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:42.915943 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:42.915907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:42.916547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:42.916046 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4"] Oct 30 04:05:43.415647 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:43.415610 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:43.516263 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:43.516232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:43.943697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:43.943650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:44.271892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:44.271859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:44.374275 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:44.374242 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:45.171258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:45.171227 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:46.688102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:46.688067 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:48.307369 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:48.307335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:49.012081 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.012045 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:49.111887 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.111842 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:49.582024 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.581979 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" podUID="c80b1e04-c613-49cf-944f-b46f80aa88cf" containerName="main" containerID="cri-o://32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c" gracePeriod=45 Oct 30 04:05:49.876260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.876233 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:05:49.967259 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967222 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-dshm\") pod \"c80b1e04-c613-49cf-944f-b46f80aa88cf\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " Oct 30 04:05:49.967463 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967277 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c80b1e04-c613-49cf-944f-b46f80aa88cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/c80b1e04-c613-49cf-944f-b46f80aa88cf-tls-certs\") pod \"c80b1e04-c613-49cf-944f-b46f80aa88cf\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " Oct 30 04:05:49.967463 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967346 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-home\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-home\") pod \"c80b1e04-c613-49cf-944f-b46f80aa88cf\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " Oct 30 04:05:49.967581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967501 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-model-cache\") pod \"c80b1e04-c613-49cf-944f-b46f80aa88cf\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " Oct 30 04:05:49.967581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967570 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80b1e04-c613-49cf-944f-b46f80aa88cf-kube-api-access-f4nvt\" (UniqueName: \"kubernetes.io/projected/c80b1e04-c613-49cf-944f-b46f80aa88cf-kube-api-access-f4nvt\") pod \"c80b1e04-c613-49cf-944f-b46f80aa88cf\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " Oct 30 04:05:49.967685 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967628 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-home" pod "c80b1e04-c613-49cf-944f-b46f80aa88cf" (UID: "c80b1e04-c613-49cf-944f-b46f80aa88cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.967790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967726 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-model-cache" pod "c80b1e04-c613-49cf-944f-b46f80aa88cf" (UID: "c80b1e04-c613-49cf-944f-b46f80aa88cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.967944 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.967925 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-tmp-dir\") pod \"c80b1e04-c613-49cf-944f-b46f80aa88cf\" (UID: \"c80b1e04-c613-49cf-944f-b46f80aa88cf\") " Oct 30 04:05:49.968217 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.968197 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-tmp-dir" pod "c80b1e04-c613-49cf-944f-b46f80aa88cf" (UID: "c80b1e04-c613-49cf-944f-b46f80aa88cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.968412 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.968399 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.968456 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.968415 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.968456 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.968424 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:49.969660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.969640 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-dshm" pod "c80b1e04-c613-49cf-944f-b46f80aa88cf" (UID: "c80b1e04-c613-49cf-944f-b46f80aa88cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:49.970133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.970115 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80b1e04-c613-49cf-944f-b46f80aa88cf-tls-certs" pod "c80b1e04-c613-49cf-944f-b46f80aa88cf" (UID: "c80b1e04-c613-49cf-944f-b46f80aa88cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:49.974443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:49.974405 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80b1e04-c613-49cf-944f-b46f80aa88cf-kube-api-access-f4nvt" pod "c80b1e04-c613-49cf-944f-b46f80aa88cf" (UID: "c80b1e04-c613-49cf-944f-b46f80aa88cf"). InnerVolumeSpecName "kube-api-access-f4nvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:50.060467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.060428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:50.069508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.069480 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80b1e04-c613-49cf-944f-b46f80aa88cf-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:50.069508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.069512 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80b1e04-c613-49cf-944f-b46f80aa88cf-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:50.069701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.069528 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4nvt\" (UniqueName: \"kubernetes.io/projected/c80b1e04-c613-49cf-944f-b46f80aa88cf-kube-api-access-f4nvt\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:50.137946 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.137917 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:50.217793 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.217744 4442 generic.go:356] "Generic (PLEG): container finished" podID="c80b1e04-c613-49cf-944f-b46f80aa88cf" containerID="32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c" exitCode=0 Oct 30 04:05:50.217986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.217847 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" Oct 30 04:05:50.217986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.217885 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" event={"ID":"c80b1e04-c613-49cf-944f-b46f80aa88cf","Type":"ContainerDied","Data":"32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c"} Oct 30 04:05:50.217986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.217932 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9" event={"ID":"c80b1e04-c613-49cf-944f-b46f80aa88cf","Type":"ContainerDied","Data":"392cc7aab5f2b7f2d807950487a1b974c888f27686cbd16fe31b600c2af23896"} Oct 30 04:05:50.217986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.217955 4442 scope.go:122] "RemoveContainer" containerID="32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c" Oct 30 04:05:50.228382 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.228360 4442 scope.go:122] "RemoveContainer" containerID="32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c" Oct 30 04:05:50.228653 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:05:50.228632 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c\": container with ID starting with 32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c not found: ID does not exist" containerID="32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c" Oct 30 04:05:50.228706 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.228663 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c"} err="failed to get container status \"32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c\": rpc error: code = NotFound desc = could not find container \"32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c\": container with ID starting with 32d6b1054e9048add157f26dfc04a248ef82bfca600a04fde72121fa92ab7b2c not found: ID does not exist" Oct 30 04:05:50.239161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.239134 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:50.265793 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.265741 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9"] Oct 30 04:05:50.268718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:50.268699 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e1a0c2/sub-test-model-e1a0c2-kserve-6754fbfc97-rnhq9"] Oct 30 04:05:51.502020 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:51.501991 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:52.107984 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:52.107947 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80b1e04-c613-49cf-944f-b46f80aa88cf" path="/var/lib/kubelet/pods/c80b1e04-c613-49cf-944f-b46f80aa88cf/volumes" Oct 30 04:05:52.245231 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:52.245197 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:52.862544 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:52.862514 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:53.281553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:53.281522 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:53.402995 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:53.402961 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:54.042487 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:54.042453 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:54.862012 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:05:54.861972 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:05:55.066205 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:55.066176 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:55.549817 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:55.549788 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:56.848965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:56.848930 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:56.947167 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:56.947132 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:57.206433 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.206350 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:57.479448 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.479351 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:57.533802 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.533768 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc"] Oct 30 04:05:57.535499 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.535467 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c80b1e04-c613-49cf-944f-b46f80aa88cf" containerName="main" Oct 30 04:05:57.535685 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.535670 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80b1e04-c613-49cf-944f-b46f80aa88cf" containerName="main" Oct 30 04:05:57.536256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.536236 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80b1e04-c613-49cf-944f-b46f80aa88cf" containerName="main" Oct 30 04:05:57.536391 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.536379 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="388c5dc2-6e38-4f8d-a4b4-6fdf5e61951f" containerName="istio-proxy" Oct 30 04:05:57.543834 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.543814 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.564306 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.564278 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc"] Oct 30 04:05:57.570746 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.570708 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-547a78-openshift-default-dockercfg-qz5cs\"" Oct 30 04:05:57.580843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.580808 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:57.671801 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.671730 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-data\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.671801 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.671801 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clkq6\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-kube-api-access-clkq6\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.672093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.671917 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-podinfo\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.672093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.671971 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-credential-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.672093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.672050 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30726b9-6954-4842-8ce1-c7c20fc326f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.672093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.672086 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-token\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.672336 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.672155 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.672336 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.672176 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-certs\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.672336 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.672283 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-envoy\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775375 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775333 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-envoy\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775445 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-data\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775504 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clkq6\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-kube-api-access-clkq6\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775661 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775572 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-podinfo\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775661 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775631 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-credential-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775705 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30726b9-6954-4842-8ce1-c7c20fc326f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775766 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-token\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775902 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775824 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.775902 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.775853 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-certs\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.776331 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.776306 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-certs\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.779212 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.779182 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-envoy\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.779466 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.779443 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-data\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.781189 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.781148 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-credential-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.781426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.781396 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-socket\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.781805 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.781729 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30726b9-6954-4842-8ce1-c7c20fc326f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.782791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.782731 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-podinfo\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.800172 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.800143 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clkq6\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-kube-api-access-clkq6\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.800290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.800256 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-token\") pod \"e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.858886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.858837 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:57.949489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.949378 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" podUID="f0cc4aba-fc20-4548-b892-941ec5d42570" containerName="main" containerID="cri-o://fa17ef49969f172efd7809a91dd372ead3539c219d649aa7aea7089a641d57e4" gracePeriod=45 Oct 30 04:05:57.989849 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:57.987506 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:58.271164 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.271126 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc"] Oct 30 04:05:58.275823 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:05:58.275791 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30726b9_6954_4842_8ce1_c7c20fc326f1.slice/crio-c8fb4c24270f7e842c8ddb1aad3120b546734576b53a52edf01296e4cec451dc WatchSource:0}: Error finding container c8fb4c24270f7e842c8ddb1aad3120b546734576b53a52edf01296e4cec451dc: Status 404 returned error can't find the container with id c8fb4c24270f7e842c8ddb1aad3120b546734576b53a52edf01296e4cec451dc Oct 30 04:05:58.280231 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.280192 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:58.280345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.280271 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:58.280345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.280312 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:05:58.317006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.316962 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:58.318516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.318485 4442 generic.go:356] "Generic (PLEG): container finished" podID="f0cc4aba-fc20-4548-b892-941ec5d42570" containerID="fa17ef49969f172efd7809a91dd372ead3539c219d649aa7aea7089a641d57e4" exitCode=0 Oct 30 04:05:58.318637 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.318528 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" event={"ID":"f0cc4aba-fc20-4548-b892-941ec5d42570","Type":"ContainerDied","Data":"fa17ef49969f172efd7809a91dd372ead3539c219d649aa7aea7089a641d57e4"} Oct 30 04:05:58.320806 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.320781 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" event={"ID":"d30726b9-6954-4842-8ce1-c7c20fc326f1","Type":"ContainerStarted","Data":"c8fb4c24270f7e842c8ddb1aad3120b546734576b53a52edf01296e4cec451dc"} Oct 30 04:05:58.403471 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.402729 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:05:58.499054 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.499024 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-home\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-home\") pod \"f0cc4aba-fc20-4548-b892-941ec5d42570\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " Oct 30 04:05:58.499240 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.499109 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0cc4aba-fc20-4548-b892-941ec5d42570-kube-api-access-nfwfq\" (UniqueName: \"kubernetes.io/projected/f0cc4aba-fc20-4548-b892-941ec5d42570-kube-api-access-nfwfq\") pod \"f0cc4aba-fc20-4548-b892-941ec5d42570\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " Oct 30 04:05:58.499240 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.499185 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-dshm\") pod \"f0cc4aba-fc20-4548-b892-941ec5d42570\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " Oct 30 04:05:58.499370 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.499354 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-model-cache\") pod \"f0cc4aba-fc20-4548-b892-941ec5d42570\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " Oct 30 04:05:58.499428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.499401 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0cc4aba-fc20-4548-b892-941ec5d42570-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0cc4aba-fc20-4548-b892-941ec5d42570-tls-certs\") pod \"f0cc4aba-fc20-4548-b892-941ec5d42570\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " Oct 30 04:05:58.499533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.499519 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-tmp-dir\") pod \"f0cc4aba-fc20-4548-b892-941ec5d42570\" (UID: \"f0cc4aba-fc20-4548-b892-941ec5d42570\") " Oct 30 04:05:58.500489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.500465 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-model-cache" pod "f0cc4aba-fc20-4548-b892-941ec5d42570" (UID: "f0cc4aba-fc20-4548-b892-941ec5d42570"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:58.500910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.500842 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-tmp-dir" pod "f0cc4aba-fc20-4548-b892-941ec5d42570" (UID: "f0cc4aba-fc20-4548-b892-941ec5d42570"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:58.500910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.500968 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-home" pod "f0cc4aba-fc20-4548-b892-941ec5d42570" (UID: "f0cc4aba-fc20-4548-b892-941ec5d42570"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:58.502604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.502580 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-dshm" pod "f0cc4aba-fc20-4548-b892-941ec5d42570" (UID: "f0cc4aba-fc20-4548-b892-941ec5d42570"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:05:58.503131 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.503114 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0cc4aba-fc20-4548-b892-941ec5d42570-kube-api-access-nfwfq" pod "f0cc4aba-fc20-4548-b892-941ec5d42570" (UID: "f0cc4aba-fc20-4548-b892-941ec5d42570"). InnerVolumeSpecName "kube-api-access-nfwfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:05:58.504133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.504113 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0cc4aba-fc20-4548-b892-941ec5d42570-tls-certs" pod "f0cc4aba-fc20-4548-b892-941ec5d42570" (UID: "f0cc4aba-fc20-4548-b892-941ec5d42570"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:05:58.601445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.601357 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:58.601445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.601399 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:58.601445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.601416 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfwfq\" (UniqueName: \"kubernetes.io/projected/f0cc4aba-fc20-4548-b892-941ec5d42570-kube-api-access-nfwfq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:58.601445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.601430 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:58.601445 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.601444 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0cc4aba-fc20-4548-b892-941ec5d42570-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:58.601743 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.601457 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0cc4aba-fc20-4548-b892-941ec5d42570-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:05:58.730343 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.729153 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v"] Oct 30 04:05:58.730979 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.730946 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" containerID="cri-o://149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d" gracePeriod=30 Oct 30 04:05:58.748905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.748871 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:58.879918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:58.876314 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:59.254219 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.254177 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:59.398778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.396378 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" event={"ID":"f0cc4aba-fc20-4548-b892-941ec5d42570","Type":"ContainerDied","Data":"6d3e7a5b2ce2191bd509dd4a061ae2d8af6a665d4e77b717bff8e341e6a1d35f"} Oct 30 04:05:59.398778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.396474 4442 scope.go:122] "RemoveContainer" containerID="fa17ef49969f172efd7809a91dd372ead3539c219d649aa7aea7089a641d57e4" Oct 30 04:05:59.398778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.396727 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4" Oct 30 04:05:59.410324 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.409556 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" event={"ID":"d30726b9-6954-4842-8ce1-c7c20fc326f1","Type":"ContainerStarted","Data":"515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6"} Oct 30 04:05:59.458971 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.458907 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podStartSLOduration=2.458888159 podStartE2EDuration="2.458888159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:05:59.457882088 +0000 UTC m=+3783.460235285" watchObservedRunningTime="2026-10-30 04:05:59.458888159 +0000 UTC m=+3783.461241357" Oct 30 04:05:59.509010 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.508866 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4"] Oct 30 04:05:59.513159 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.513129 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-547a78/sub-test-model-547a78-kserve-5f5b95d45-dc2s4"] Oct 30 04:05:59.551810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.551783 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:59.859655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.859566 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:05:59.861545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.861520 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 04:05:59.861683 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:05:59.861572 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 04:06:00.115366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:00.115280 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0cc4aba-fc20-4548-b892-941ec5d42570" path="/var/lib/kubelet/pods/f0cc4aba-fc20-4548-b892-941ec5d42570/volumes" Oct 30 04:06:00.875778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:00.868273 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 04:06:00.875778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:00.868354 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 04:06:01.375938 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:01.375903 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:01.859533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:01.859485 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 04:06:01.859704 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:01.859579 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 04:06:02.859840 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:02.859803 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 04:06:02.860241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:02.859859 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 04:06:03.062904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:03.062869 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:03.860168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:03.860133 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 04:06:03.860591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:03.860196 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 04:06:04.055190 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.055166 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 04:06:04.149251 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.149218 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-certs\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.149508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.149489 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-socket\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.149682 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.149488 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-certs" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:04.149783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.149664 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-credential-socket\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.149783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.149653 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-socket" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:04.149921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.149787 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-istio-token\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-istio-token\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.149921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.149906 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-kube-api-access-tgxl7\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-kube-api-access-tgxl7\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.150251 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.150229 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-credential-socket" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:04.150376 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.150317 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8622786e-9c38-468e-8da1-6522e412e5b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622786e-9c38-468e-8da1-6522e412e5b0-istiod-ca-cert\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.150650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.150631 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-data\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.150811 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.150795 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8622786e-9c38-468e-8da1-6522e412e5b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622786e-9c38-468e-8da1-6522e412e5b0-istio-podinfo\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.150935 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.150896 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-data" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:04.151031 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.150941 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-envoy\") pod \"8622786e-9c38-468e-8da1-6522e412e5b0\" (UID: \"8622786e-9c38-468e-8da1-6522e412e5b0\") " Oct 30 04:06:04.151627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.151579 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.151627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.151604 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.151627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.151619 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.151969 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.151632 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.151969 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.151705 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8622786e-9c38-468e-8da1-6522e412e5b0-istiod-ca-cert" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:04.153258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.153223 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-kube-api-access-tgxl7" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "kube-api-access-tgxl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:04.153576 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.153536 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-istio-token" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:04.156636 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.154453 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8622786e-9c38-468e-8da1-6522e412e5b0-istio-podinfo" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:06:04.161428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.161396 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-envoy" pod "8622786e-9c38-468e-8da1-6522e412e5b0" (UID: "8622786e-9c38-468e-8da1-6522e412e5b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:04.218461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.218389 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:04.252983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.252951 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.252983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.252980 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgxl7\" (UniqueName: \"kubernetes.io/projected/8622786e-9c38-468e-8da1-6522e412e5b0-kube-api-access-tgxl7\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.252983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.252990 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8622786e-9c38-468e-8da1-6522e412e5b0-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.253255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.252998 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8622786e-9c38-468e-8da1-6522e412e5b0-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.253255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.253007 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8622786e-9c38-468e-8da1-6522e412e5b0-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:04.318324 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.318226 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:04.448677 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.448639 4442 generic.go:356] "Generic (PLEG): container finished" podID="8622786e-9c38-468e-8da1-6522e412e5b0" containerID="149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d" exitCode=0 Oct 30 04:06:04.448894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.448694 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" Oct 30 04:06:04.448894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.448729 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" event={"ID":"8622786e-9c38-468e-8da1-6522e412e5b0","Type":"ContainerDied","Data":"149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d"} Oct 30 04:06:04.448894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.448804 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v" event={"ID":"8622786e-9c38-468e-8da1-6522e412e5b0","Type":"ContainerDied","Data":"fd1e7e43049a742e4a0e1390dd143228656c61035aa527426a025f4371a006d2"} Oct 30 04:06:04.448894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.448829 4442 scope.go:122] "RemoveContainer" containerID="149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d" Oct 30 04:06:04.462107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.462088 4442 scope.go:122] "RemoveContainer" containerID="b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14" Oct 30 04:06:04.472241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.472220 4442 scope.go:122] "RemoveContainer" containerID="149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d" Oct 30 04:06:04.472541 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:04.472515 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d\": container with ID starting with 149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d not found: ID does not exist" containerID="149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d" Oct 30 04:06:04.472611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.472552 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d"} err="failed to get container status \"149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d\": rpc error: code = NotFound desc = could not find container \"149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d\": container with ID starting with 149d735a9574ad3465eae5f9eb7494dd696343fed6f11cacec07fc24aa315e1d not found: ID does not exist" Oct 30 04:06:04.472611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.472576 4442 scope.go:122] "RemoveContainer" containerID="b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14" Oct 30 04:06:04.472888 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:04.472867 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14\": container with ID starting with b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14 not found: ID does not exist" containerID="b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14" Oct 30 04:06:04.472972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.472893 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14"} err="failed to get container status \"b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14\": rpc error: code = NotFound desc = could not find container \"b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14\": container with ID starting with b74004315b8d1b5b9f736757f27dda0f48dbeb514118250e49371503baa63c14 not found: ID does not exist" Oct 30 04:06:04.516878 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.516838 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v"] Oct 30 04:06:04.520091 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.520066 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-81afe3-openshift-default-9c5c4c49-2sv2v"] Oct 30 04:06:04.860595 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.860136 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 04:06:04.860595 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.860196 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 04:06:04.872285 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:04.872151 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:05.865024 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:05.864990 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:06:05.865569 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:05.865545 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:06:05.866435 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:05.866418 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:06:06.122348 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:06.122271 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" path="/var/lib/kubelet/pods/8622786e-9c38-468e-8da1-6522e412e5b0/volumes" Oct 30 04:06:06.687414 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:06.687380 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:07.138187 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:07.138153 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:07.451159 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:07.451086 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf"] Oct 30 04:06:07.462674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:07.462646 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:07.490457 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:07.490419 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t"] Oct 30 04:06:07.589204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:07.584054 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:07.703864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:07.702342 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:08.129802 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:08.127918 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:08.700656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:08.700585 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:09.443075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:09.443043 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:10.889912 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:10.889862 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:12.407451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:12.407416 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:12.451431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:12.451392 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" podUID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" containerName="payload-processing" containerID="cri-o://513ab0611dfcf1a02d96ed4ebe7194c5ec2a67aed8e91bc55da6769bad991a88" gracePeriod=25 Oct 30 04:06:12.492903 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:12.492822 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" podUID="891f78f5-3792-4618-904b-2e1a6ae804cc" containerName="payload-pre-processing" containerID="cri-o://de2902ba476da8505e934c4411ee62bda97fdf792d4629b24d0f4ffe169d9bbd" gracePeriod=25 Oct 30 04:06:13.657072 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:13.657039 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:13.758057 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:13.758004 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:13.836862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:13.836832 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:14.000037 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:13.999984 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:14.441816 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:14.441572 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:14.824104 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:14.824076 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:15.642860 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:15.642811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:15.748262 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:15.748232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:16.006106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:16.006076 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:16.106886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:16.106856 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:16.713144 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:16.713111 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:17.126971 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:17.126941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:17.500633 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:17.500602 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:17.587833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:17.587806 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:17.889014 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:17.888915 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:19.202880 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:19.202845 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:19.822535 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:19.822503 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:06:20.297339 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:20.297302 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" podUID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:20.297895 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:20.297319 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" podUID="891f78f5-3792-4618-904b-2e1a6ae804cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:20.908901 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:20.908859 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:20.915693 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:20.915659 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn"] Oct 30 04:06:21.054535 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:21.054495 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:21.155409 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:21.155372 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:22.176366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:22.176318 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:22.690900 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:22.690861 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:23.616875 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:23.616839 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j"] Oct 30 04:06:23.743699 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:23.743664 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:23.843819 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:23.843781 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:24.430194 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:24.430163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:24.630652 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:24.630618 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:24.861684 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:24.861650 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:06:25.916323 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:25.916274 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" podUID="7e4351d9-3c06-4da5-96a8-11ffd833984a" containerName="payload-processing" containerID="cri-o://bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00" gracePeriod=25 Oct 30 04:06:26.135082 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:26.135041 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:27.153603 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:27.153564 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" podUID="7e4351d9-3c06-4da5-96a8-11ffd833984a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:27.780263 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:27.780224 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d"] Oct 30 04:06:27.780491 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:27.780465 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" podUID="dd79d028-431b-43e4-bc99-e42209568e78" containerName="maas-api" containerID="cri-o://ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418" gracePeriod=30 Oct 30 04:06:27.824732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:27.824696 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:28.137598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.137569 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 04:06:28.299833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.268369 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd79d028-431b-43e4-bc99-e42209568e78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd79d028-431b-43e4-bc99-e42209568e78-openshift-service-ca\") pod \"dd79d028-431b-43e4-bc99-e42209568e78\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " Oct 30 04:06:28.300481 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.300443 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-metrics-certs\") pod \"dd79d028-431b-43e4-bc99-e42209568e78\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " Oct 30 04:06:28.301153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.301073 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd79d028-431b-43e4-bc99-e42209568e78-openshift-service-ca" pod "dd79d028-431b-43e4-bc99-e42209568e78" (UID: "dd79d028-431b-43e4-bc99-e42209568e78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:28.308991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.308965 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd79d028-431b-43e4-bc99-e42209568e78-kube-api-access-zm9h2\" (UniqueName: \"kubernetes.io/projected/dd79d028-431b-43e4-bc99-e42209568e78-kube-api-access-zm9h2\") pod \"dd79d028-431b-43e4-bc99-e42209568e78\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " Oct 30 04:06:28.309253 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.309234 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-maas-api-tls\") pod \"dd79d028-431b-43e4-bc99-e42209568e78\" (UID: \"dd79d028-431b-43e4-bc99-e42209568e78\") " Oct 30 04:06:28.328536 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.321966 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-metrics-certs" pod "dd79d028-431b-43e4-bc99-e42209568e78" (UID: "dd79d028-431b-43e4-bc99-e42209568e78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:28.328536 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.322686 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:28.328536 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.326428 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd79d028-431b-43e4-bc99-e42209568e78-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:28.328536 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.326452 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:28.331322 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.331274 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd79d028-431b-43e4-bc99-e42209568e78-kube-api-access-zm9h2" pod "dd79d028-431b-43e4-bc99-e42209568e78" (UID: "dd79d028-431b-43e4-bc99-e42209568e78"). InnerVolumeSpecName "kube-api-access-zm9h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:28.331611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.331586 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-maas-api-tls" pod "dd79d028-431b-43e4-bc99-e42209568e78" (UID: "dd79d028-431b-43e4-bc99-e42209568e78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:06:28.427617 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.427587 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9h2\" (UniqueName: \"kubernetes.io/projected/dd79d028-431b-43e4-bc99-e42209568e78-kube-api-access-zm9h2\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:28.427617 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.427614 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd79d028-431b-43e4-bc99-e42209568e78-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:28.618844 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.617635 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" podUID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" containerName="payload-processing" containerID="cri-o://045466d43cdeb20d275fa8a1b7c4c2696422af83c2b94fbcff5c3d7cd2187dfa" gracePeriod=25 Oct 30 04:06:28.632238 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.632198 4442 generic.go:356] "Generic (PLEG): container finished" podID="dd79d028-431b-43e4-bc99-e42209568e78" containerID="ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418" exitCode=0 Oct 30 04:06:28.632431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.632308 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" Oct 30 04:06:28.632431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.632409 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" event={"ID":"dd79d028-431b-43e4-bc99-e42209568e78","Type":"ContainerDied","Data":"ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418"} Oct 30 04:06:28.632595 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.632469 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d" event={"ID":"dd79d028-431b-43e4-bc99-e42209568e78","Type":"ContainerDied","Data":"49cbc14c1505414601504e5933d13c3a6cf04acfb8768249bc63e1271669d91f"} Oct 30 04:06:28.632595 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.632491 4442 scope.go:122] "RemoveContainer" containerID="ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418" Oct 30 04:06:28.646688 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.646662 4442 scope.go:122] "RemoveContainer" containerID="ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418" Oct 30 04:06:28.647056 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:28.646992 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418\": container with ID starting with ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418 not found: ID does not exist" containerID="ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418" Oct 30 04:06:28.647056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.647035 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418"} err="failed to get container status \"ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418\": rpc error: code = NotFound desc = could not find container \"ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418\": container with ID starting with ca707a9e9be15e63c53577e89f6567f29c9a79737d804ff72de8f6101be19418 not found: ID does not exist" Oct 30 04:06:28.684177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.684150 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d"] Oct 30 04:06:28.687022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:28.686997 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-1a1b45-cbd8b84c8-cdh7d"] Oct 30 04:06:29.147911 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:29.147878 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:30.105554 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:30.105519 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:30.118531 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:30.118490 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd79d028-431b-43e4-bc99-e42209568e78" path="/var/lib/kubelet/pods/dd79d028-431b-43e4-bc99-e42209568e78/volumes" Oct 30 04:06:30.297530 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:30.297479 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" podUID="891f78f5-3792-4618-904b-2e1a6ae804cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:30.297530 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:30.297529 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" podUID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:30.928990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:30.928955 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:31.247406 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:31.247326 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:31.347556 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:31.347524 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:31.983467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:31.983435 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:32.416787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.416739 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:32.664790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.662936 4442 generic.go:356] "Generic (PLEG): container finished" podID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" containerID="513ab0611dfcf1a02d96ed4ebe7194c5ec2a67aed8e91bc55da6769bad991a88" exitCode=0 Oct 30 04:06:32.664790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.663085 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" event={"ID":"ac7523a6-6cad-44c1-9ae8-0005ea2ae558","Type":"ContainerDied","Data":"513ab0611dfcf1a02d96ed4ebe7194c5ec2a67aed8e91bc55da6769bad991a88"} Oct 30 04:06:32.664790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.664719 4442 generic.go:356] "Generic (PLEG): container finished" podID="891f78f5-3792-4618-904b-2e1a6ae804cc" containerID="de2902ba476da8505e934c4411ee62bda97fdf792d4629b24d0f4ffe169d9bbd" exitCode=0 Oct 30 04:06:32.665080 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.664834 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" event={"ID":"891f78f5-3792-4618-904b-2e1a6ae804cc","Type":"ContainerDied","Data":"de2902ba476da8505e934c4411ee62bda97fdf792d4629b24d0f4ffe169d9bbd"} Oct 30 04:06:32.722472 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.722440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:32.811798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.811775 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 04:06:32.844718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.844692 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 04:06:32.878385 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.878358 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:32.879387 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.879357 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891f78f5-3792-4618-904b-2e1a6ae804cc-kube-api-access-ncr8m\" (UniqueName: \"kubernetes.io/projected/891f78f5-3792-4618-904b-2e1a6ae804cc-kube-api-access-ncr8m\") pod \"891f78f5-3792-4618-904b-2e1a6ae804cc\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " Oct 30 04:06:32.879532 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.879445 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-kube-api-access-pr5zp\" (UniqueName: \"kubernetes.io/projected/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-kube-api-access-pr5zp\") pod \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " Oct 30 04:06:32.879619 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.879571 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-plugins-config-volume\") pod \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\" (UID: \"ac7523a6-6cad-44c1-9ae8-0005ea2ae558\") " Oct 30 04:06:32.879683 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.879639 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891f78f5-3792-4618-904b-2e1a6ae804cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891f78f5-3792-4618-904b-2e1a6ae804cc-plugins-config-volume\") pod \"891f78f5-3792-4618-904b-2e1a6ae804cc\" (UID: \"891f78f5-3792-4618-904b-2e1a6ae804cc\") " Oct 30 04:06:32.880067 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.880040 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-plugins-config-volume" pod "ac7523a6-6cad-44c1-9ae8-0005ea2ae558" (UID: "ac7523a6-6cad-44c1-9ae8-0005ea2ae558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:32.880219 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.880039 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891f78f5-3792-4618-904b-2e1a6ae804cc-plugins-config-volume" pod "891f78f5-3792-4618-904b-2e1a6ae804cc" (UID: "891f78f5-3792-4618-904b-2e1a6ae804cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:32.880360 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.880336 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:32.880505 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.880364 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/891f78f5-3792-4618-904b-2e1a6ae804cc-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:32.882012 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.881983 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891f78f5-3792-4618-904b-2e1a6ae804cc-kube-api-access-ncr8m" pod "891f78f5-3792-4618-904b-2e1a6ae804cc" (UID: "891f78f5-3792-4618-904b-2e1a6ae804cc"). InnerVolumeSpecName "kube-api-access-ncr8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:32.882394 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.882371 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-kube-api-access-pr5zp" pod "ac7523a6-6cad-44c1-9ae8-0005ea2ae558" (UID: "ac7523a6-6cad-44c1-9ae8-0005ea2ae558"). InnerVolumeSpecName "kube-api-access-pr5zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:32.982227 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.982140 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncr8m\" (UniqueName: \"kubernetes.io/projected/891f78f5-3792-4618-904b-2e1a6ae804cc-kube-api-access-ncr8m\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:32.982227 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:32.982176 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5zp\" (UniqueName: \"kubernetes.io/projected/ac7523a6-6cad-44c1-9ae8-0005ea2ae558-kube-api-access-pr5zp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:33.320416 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.320387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:33.672991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.672914 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" Oct 30 04:06:33.672991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.672925 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t" event={"ID":"891f78f5-3792-4618-904b-2e1a6ae804cc","Type":"ContainerDied","Data":"0bffeeaaf6ad92094b7b86ceb9eb7a81413c967d526ae229df362a2c7b7f12a8"} Oct 30 04:06:33.672991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.672987 4442 scope.go:122] "RemoveContainer" containerID="de2902ba476da8505e934c4411ee62bda97fdf792d4629b24d0f4ffe169d9bbd" Oct 30 04:06:33.675537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.675509 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" event={"ID":"ac7523a6-6cad-44c1-9ae8-0005ea2ae558","Type":"ContainerDied","Data":"8c3378c8dad59547e86f577b846fe9de2ea567e2c2faf03f0669b37f439e772e"} Oct 30 04:06:33.675639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.675613 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf" Oct 30 04:06:33.683563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.683541 4442 scope.go:122] "RemoveContainer" containerID="513ab0611dfcf1a02d96ed4ebe7194c5ec2a67aed8e91bc55da6769bad991a88" Oct 30 04:06:33.727210 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.727177 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t"] Oct 30 04:06:33.738179 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.738156 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6ad168-848c96b775-wj48t"] Oct 30 04:06:33.774543 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.774513 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf"] Oct 30 04:06:33.782716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:33.782687 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6ad168-7b664c97b4-cr7bf"] Oct 30 04:06:34.108059 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:34.108021 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891f78f5-3792-4618-904b-2e1a6ae804cc" path="/var/lib/kubelet/pods/891f78f5-3792-4618-904b-2e1a6ae804cc/volumes" Oct 30 04:06:34.108506 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:34.108486 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" path="/var/lib/kubelet/pods/ac7523a6-6cad-44c1-9ae8-0005ea2ae558/volumes" Oct 30 04:06:34.956129 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:34.956057 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:36.391524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:36.391492 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:36.787241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:36.787214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:37.015247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:37.015218 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:37.153397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:37.153301 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" podUID="7e4351d9-3c06-4da5-96a8-11ffd833984a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:37.458966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:37.458881 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:37.654973 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:37.654932 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" podUID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:37.820956 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:37.820919 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8"] Oct 30 04:06:37.821215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:37.821186 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" containerID="cri-o://75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2" gracePeriod=30 Oct 30 04:06:37.845428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:37.845399 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:38.175314 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:38.175229 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:38.758215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:38.758178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:39.998870 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:39.998809 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:40.285540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:40.285463 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml"] Oct 30 04:06:40.424059 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:40.424025 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:40.533467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:40.533381 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:40.877171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:40.877129 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:41.619179 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:41.619134 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:42.887073 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.887037 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9"] Oct 30 04:06:42.888054 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888031 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" Oct 30 04:06:42.888112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888057 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" Oct 30 04:06:42.888112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888080 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" Oct 30 04:06:42.888112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888085 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" Oct 30 04:06:42.888112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888108 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f0cc4aba-fc20-4548-b892-941ec5d42570" containerName="main" Oct 30 04:06:42.888112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888114 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0cc4aba-fc20-4548-b892-941ec5d42570" containerName="main" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888133 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" containerName="payload-processing" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888137 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" containerName="payload-processing" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888151 4442 cpu_manager.go:414] "containerMap: removing container" podUID="891f78f5-3792-4618-904b-2e1a6ae804cc" containerName="payload-pre-processing" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888157 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891f78f5-3792-4618-904b-2e1a6ae804cc" containerName="payload-pre-processing" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888165 4442 cpu_manager.go:414] "containerMap: removing container" podUID="dd79d028-431b-43e4-bc99-e42209568e78" containerName="maas-api" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888169 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd79d028-431b-43e4-bc99-e42209568e78" containerName="maas-api" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888265 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7523a6-6cad-44c1-9ae8-0005ea2ae558" containerName="payload-processing" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888276 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="891f78f5-3792-4618-904b-2e1a6ae804cc" containerName="payload-pre-processing" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888282 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd79d028-431b-43e4-bc99-e42209568e78" containerName="maas-api" Oct 30 04:06:42.888290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888289 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" Oct 30 04:06:42.888600 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888297 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8622786e-9c38-468e-8da1-6522e412e5b0" containerName="istio-proxy" Oct 30 04:06:42.888600 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.888305 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0cc4aba-fc20-4548-b892-941ec5d42570" containerName="main" Oct 30 04:06:42.891807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.891790 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.905376 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.905354 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-7d99a4-openshift-default-dockercfg-wfkbf\"" Oct 30 04:06:42.922537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.922504 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9"] Oct 30 04:06:42.934890 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.934857 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935061 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.934906 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935126 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.935107 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935187 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.935165 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935249 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.935220 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.935300 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6x4\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-kube-api-access-6m6x4\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935385 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.935359 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935441 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.935408 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:42.935500 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:42.935444 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037285 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6x4\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-kube-api-access-6m6x4\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037365 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037421 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037457 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037494 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037527 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037682 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037728 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.037993 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.037811 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.039716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.038220 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-data\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.039716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.038485 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-credential-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.039716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.038729 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-socket\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.039716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.038910 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istiod-ca-cert\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.039716 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.039145 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-certs\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.041709 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.041568 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-podinfo\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.041709 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.041593 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-envoy\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.062332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.062304 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6x4\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-kube-api-access-6m6x4\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.065384 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.065341 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-token\") pod \"e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.100383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.100327 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:06:43.115137 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.115110 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:43.138527 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138501 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-socket\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.138620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138539 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-kube-api-access-nc9lq\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-kube-api-access-nc9lq\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.138620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138611 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-envoy\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.138737 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138629 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-token\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-token\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.138737 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138661 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/723efe3e-62f7-48ba-b1ed-c15032afd156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723efe3e-62f7-48ba-b1ed-c15032afd156-istiod-ca-cert\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.138878 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138769 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-socket" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:43.138878 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138747 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-certs\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.138991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138879 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-podinfo\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.139042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.138954 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-data\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.139165 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139050 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-certs" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:43.139165 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139063 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-credential-socket\") pod \"723efe3e-62f7-48ba-b1ed-c15032afd156\" (UID: \"723efe3e-62f7-48ba-b1ed-c15032afd156\") " Oct 30 04:06:43.139165 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139124 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/723efe3e-62f7-48ba-b1ed-c15032afd156-istiod-ca-cert" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:43.139324 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139260 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-data" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:43.139549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139412 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-credential-socket" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:43.139613 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139593 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.139672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139612 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.139672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139625 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.139672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139634 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/723efe3e-62f7-48ba-b1ed-c15032afd156-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.139672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.139641 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.141199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.141172 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-kube-api-access-nc9lq" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "kube-api-access-nc9lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:43.141308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.141284 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-podinfo" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:06:43.141669 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.141600 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-envoy" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:43.141669 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.141630 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-token" pod "723efe3e-62f7-48ba-b1ed-c15032afd156" (UID: "723efe3e-62f7-48ba-b1ed-c15032afd156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:43.230946 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.230915 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:43.241086 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.241063 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.241086 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.241086 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.241250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.241095 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/723efe3e-62f7-48ba-b1ed-c15032afd156-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.241250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.241105 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9lq\" (UniqueName: \"kubernetes.io/projected/723efe3e-62f7-48ba-b1ed-c15032afd156-kube-api-access-nc9lq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:43.549527 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.549497 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9"] Oct 30 04:06:43.550410 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:06:43.550385 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f831b01_41b2_4f28_b2b4_b33568d8c70e.slice/crio-eb00fb4a60f1c482a05203fdd1da7f2ab8585ce6a9694c0758b785bf043db3d0 WatchSource:0}: Error finding container eb00fb4a60f1c482a05203fdd1da7f2ab8585ce6a9694c0758b785bf043db3d0: Status 404 returned error can't find the container with id eb00fb4a60f1c482a05203fdd1da7f2ab8585ce6a9694c0758b785bf043db3d0 Oct 30 04:06:43.552946 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.552913 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:06:43.553021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.552982 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:06:43.553021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.553009 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:06:43.748317 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.748283 4442 generic.go:356] "Generic (PLEG): container finished" podID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerID="75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2" exitCode=0 Oct 30 04:06:43.748533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.748349 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" Oct 30 04:06:43.748533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.748387 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" event={"ID":"723efe3e-62f7-48ba-b1ed-c15032afd156","Type":"ContainerDied","Data":"75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2"} Oct 30 04:06:43.748533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.748455 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8" event={"ID":"723efe3e-62f7-48ba-b1ed-c15032afd156","Type":"ContainerDied","Data":"94b44d6173a387d59d7ccc3d2cac197764170e30d32a7c886b7676905c9552a3"} Oct 30 04:06:43.748533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.748478 4442 scope.go:122] "RemoveContainer" containerID="75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2" Oct 30 04:06:43.751208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.751181 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" event={"ID":"3f831b01-41b2-4f28-b2b4-b33568d8c70e","Type":"ContainerStarted","Data":"3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce"} Oct 30 04:06:43.751421 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.751396 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" event={"ID":"3f831b01-41b2-4f28-b2b4-b33568d8c70e","Type":"ContainerStarted","Data":"eb00fb4a60f1c482a05203fdd1da7f2ab8585ce6a9694c0758b785bf043db3d0"} Oct 30 04:06:43.762343 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.762323 4442 scope.go:122] "RemoveContainer" containerID="75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2" Oct 30 04:06:43.762631 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:43.762608 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2\": container with ID starting with 75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2 not found: ID does not exist" containerID="75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2" Oct 30 04:06:43.762702 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.762640 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2"} err="failed to get container status \"75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2\": rpc error: code = NotFound desc = could not find container \"75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2\": container with ID starting with 75203e1a753d28c2f3f3e3b61f87f92397c666b0c4e3e697f325b80c75bbbdc2 not found: ID does not exist" Oct 30 04:06:43.799427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.799354 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8"] Oct 30 04:06:43.809980 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.809948 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-1a1b45-openshift-default-67765b47b5-mj8d8"] Oct 30 04:06:43.850075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:43.850024 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podStartSLOduration=1.850009863 podStartE2EDuration="1.850009863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:06:43.847731358 +0000 UTC m=+3827.850084558" watchObservedRunningTime="2026-10-30 04:06:43.850009863 +0000 UTC m=+3827.852363060" Oct 30 04:06:44.108568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:44.108493 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" path="/var/lib/kubelet/pods/723efe3e-62f7-48ba-b1ed-c15032afd156/volumes" Oct 30 04:06:44.231318 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:44.231285 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:44.233301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:44.233264 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 04:06:44.233448 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:44.233320 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 04:06:44.440112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:44.440042 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv"] Oct 30 04:06:44.562318 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:44.562285 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:44.656438 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:44.656405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:45.231285 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:45.231248 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 04:06:45.231711 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:45.231303 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 04:06:45.306780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:45.292825 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" podUID="2c775e5d-3134-40d3-8d81-4748781896a5" containerName="payload-processing" containerID="cri-o://8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072" gracePeriod=25 Oct 30 04:06:45.543705 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:45.543668 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:45.654473 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:45.654436 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:46.231367 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.231340 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 04:06:46.231724 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.231395 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 04:06:46.237176 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.237156 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 04:06:46.327327 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.327295 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4351d9-3c06-4da5-96a8-11ffd833984a-kube-api-access-7gk8t\" (UniqueName: \"kubernetes.io/projected/7e4351d9-3c06-4da5-96a8-11ffd833984a-kube-api-access-7gk8t\") pod \"7e4351d9-3c06-4da5-96a8-11ffd833984a\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " Oct 30 04:06:46.327553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.327422 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4351d9-3c06-4da5-96a8-11ffd833984a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4351d9-3c06-4da5-96a8-11ffd833984a-plugins-config-volume\") pod \"7e4351d9-3c06-4da5-96a8-11ffd833984a\" (UID: \"7e4351d9-3c06-4da5-96a8-11ffd833984a\") " Oct 30 04:06:46.327864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.327840 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4351d9-3c06-4da5-96a8-11ffd833984a-plugins-config-volume" pod "7e4351d9-3c06-4da5-96a8-11ffd833984a" (UID: "7e4351d9-3c06-4da5-96a8-11ffd833984a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:46.329691 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.329667 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4351d9-3c06-4da5-96a8-11ffd833984a-kube-api-access-7gk8t" pod "7e4351d9-3c06-4da5-96a8-11ffd833984a" (UID: "7e4351d9-3c06-4da5-96a8-11ffd833984a"). InnerVolumeSpecName "kube-api-access-7gk8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:46.429308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.429274 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e4351d9-3c06-4da5-96a8-11ffd833984a-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:46.429308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.429300 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gk8t\" (UniqueName: \"kubernetes.io/projected/7e4351d9-3c06-4da5-96a8-11ffd833984a-kube-api-access-7gk8t\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:46.777833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.777798 4442 generic.go:356] "Generic (PLEG): container finished" podID="7e4351d9-3c06-4da5-96a8-11ffd833984a" containerID="bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00" exitCode=0 Oct 30 04:06:46.778005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.777855 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" Oct 30 04:06:46.778005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.777863 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" event={"ID":"7e4351d9-3c06-4da5-96a8-11ffd833984a","Type":"ContainerDied","Data":"bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00"} Oct 30 04:06:46.778005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.777929 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn" event={"ID":"7e4351d9-3c06-4da5-96a8-11ffd833984a","Type":"ContainerDied","Data":"39b917a88e32825d6a10dfc1f77be13dfbfc7fbf03875b9dc795c1c9508f6fa0"} Oct 30 04:06:46.778005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.777952 4442 scope.go:122] "RemoveContainer" containerID="bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00" Oct 30 04:06:46.814386 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.814367 4442 scope.go:122] "RemoveContainer" containerID="bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00" Oct 30 04:06:46.814714 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:46.814693 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00\": container with ID starting with bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00 not found: ID does not exist" containerID="bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00" Oct 30 04:06:46.814801 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.814725 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00"} err="failed to get container status \"bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00\": rpc error: code = NotFound desc = could not find container \"bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00\": container with ID starting with bd258e5a356954a6be2dc237ca9ade7de132782f7a784954fc814d05d2f26d00 not found: ID does not exist" Oct 30 04:06:46.847605 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.847581 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn"] Oct 30 04:06:46.858306 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:46.858285 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-1a1b45-6d8645dc58-2qkpn"] Oct 30 04:06:47.008786 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:47.008293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:47.209323 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:47.209243 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:47.231882 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:47.231851 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 04:06:47.232261 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:47.231908 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 04:06:47.655255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:47.655216 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" podUID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:48.049063 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.048942 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b"] Oct 30 04:06:48.070294 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.070264 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:48.123395 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.123348 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4351d9-3c06-4da5-96a8-11ffd833984a" path="/var/lib/kubelet/pods/7e4351d9-3c06-4da5-96a8-11ffd833984a/volumes" Oct 30 04:06:48.237779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.232224 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 04:06:48.237779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.232285 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 04:06:48.738156 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.737593 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:48.821296 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.821266 4442 generic.go:356] "Generic (PLEG): container finished" podID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" containerID="045466d43cdeb20d275fa8a1b7c4c2696422af83c2b94fbcff5c3d7cd2187dfa" exitCode=0 Oct 30 04:06:48.821469 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.821425 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" event={"ID":"acdb7dd7-0fa4-45e8-995e-2ec2238049c5","Type":"ContainerDied","Data":"045466d43cdeb20d275fa8a1b7c4c2696422af83c2b94fbcff5c3d7cd2187dfa"} Oct 30 04:06:48.868792 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.864251 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:48.947550 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:48.947523 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 04:06:49.012785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.012722 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:49.065958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.065918 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-kube-api-access-bhqwm\" (UniqueName: \"kubernetes.io/projected/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-kube-api-access-bhqwm\") pod \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " Oct 30 04:06:49.066224 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.066201 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-plugins-config-volume\") pod \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\" (UID: \"acdb7dd7-0fa4-45e8-995e-2ec2238049c5\") " Oct 30 04:06:49.067699 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.067665 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-plugins-config-volume" pod "acdb7dd7-0fa4-45e8-995e-2ec2238049c5" (UID: "acdb7dd7-0fa4-45e8-995e-2ec2238049c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:49.072725 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.072693 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-kube-api-access-bhqwm" pod "acdb7dd7-0fa4-45e8-995e-2ec2238049c5" (UID: "acdb7dd7-0fa4-45e8-995e-2ec2238049c5"). InnerVolumeSpecName "kube-api-access-bhqwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:49.168196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.168158 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:49.168196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.168188 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhqwm\" (UniqueName: \"kubernetes.io/projected/acdb7dd7-0fa4-45e8-995e-2ec2238049c5-kube-api-access-bhqwm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:49.231461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.231381 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 04:06:49.231461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.231429 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 04:06:49.441014 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.440966 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" podUID="2b37ede5-6f78-44aa-9211-4e23d05941c2" containerName="payload-processing" containerID="cri-o://993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7" gracePeriod=25 Oct 30 04:06:49.837788 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.837350 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:06:49.874728 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.874692 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" event={"ID":"acdb7dd7-0fa4-45e8-995e-2ec2238049c5","Type":"ContainerDied","Data":"ffdb67ff7753e7647060d481f0b075d97288091b12ff660d89a9b6c6b17e5196"} Oct 30 04:06:49.874991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.874770 4442 scope.go:122] "RemoveContainer" containerID="045466d43cdeb20d275fa8a1b7c4c2696422af83c2b94fbcff5c3d7cd2187dfa" Oct 30 04:06:49.874991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.874884 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j" Oct 30 04:06:49.934084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.934032 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j"] Oct 30 04:06:49.937580 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:49.937558 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-35f43f-8458dfc69b-sl27j"] Oct 30 04:06:50.118115 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:50.118037 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" path="/var/lib/kubelet/pods/acdb7dd7-0fa4-45e8-995e-2ec2238049c5/volumes" Oct 30 04:06:50.231443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:50.231406 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 04:06:50.231629 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:50.231467 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 04:06:50.661540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:50.661256 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" podUID="2c775e5d-3134-40d3-8d81-4748781896a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:50.714365 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:50.714333 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:51.488053 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:51.488019 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:51.488352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:51.488327 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:51.489239 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:51.489217 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:06:51.958809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:51.958774 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw"] Oct 30 04:06:51.991973 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:51.991944 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:52.383554 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:52.383527 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:52.581554 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:52.581517 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:53.116073 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:53.116044 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:53.453783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:53.453668 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:53.494382 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:53.494349 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc"] Oct 30 04:06:53.494613 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:53.494590 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" containerID="cri-o://515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6" gracePeriod=30 Oct 30 04:06:53.552970 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:53.552939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:54.007286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:54.007232 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:54.326975 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:54.326872 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:54.862630 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:54.862596 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:06:55.213479 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:55.213397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:55.696009 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:55.695972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:56.330017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:56.329988 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:57.345121 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:57.345080 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" podUID="2b37ede5-6f78-44aa-9211-4e23d05941c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:06:57.681390 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:57.681311 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:57.819060 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:57.819033 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:58.788243 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.788216 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:06:58.895273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895241 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-envoy\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895311 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-data\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895456 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-credential-socket\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895501 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-kube-api-access-clkq6\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-kube-api-access-clkq6\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895545 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-certs\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895592 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-token\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-token\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895635 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-socket\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895707 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-podinfo\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.896524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.895732 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30726b9-6954-4842-8ce1-c7c20fc326f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30726b9-6954-4842-8ce1-c7c20fc326f1-istiod-ca-cert\") pod \"d30726b9-6954-4842-8ce1-c7c20fc326f1\" (UID: \"d30726b9-6954-4842-8ce1-c7c20fc326f1\") " Oct 30 04:06:58.897453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.896682 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30726b9-6954-4842-8ce1-c7c20fc326f1-istiod-ca-cert" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:06:58.897453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.897250 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-certs" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:58.897557 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.897454 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-data" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:58.897663 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.897638 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-credential-socket" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:58.897965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.897942 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-socket" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:58.911002 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.910970 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-token" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:58.911141 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.911019 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-kube-api-access-clkq6" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "kube-api-access-clkq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:06:58.911215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.911133 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-envoy" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:06:58.914811 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.914781 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-podinfo" pod "d30726b9-6954-4842-8ce1-c7c20fc326f1" (UID: "d30726b9-6954-4842-8ce1-c7c20fc326f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:06:58.948515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.948484 4442 generic.go:356] "Generic (PLEG): container finished" podID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerID="515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6" exitCode=0 Oct 30 04:06:58.948683 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.948661 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" event={"ID":"d30726b9-6954-4842-8ce1-c7c20fc326f1","Type":"ContainerDied","Data":"515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6"} Oct 30 04:06:58.948811 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.948792 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" event={"ID":"d30726b9-6954-4842-8ce1-c7c20fc326f1","Type":"ContainerDied","Data":"c8fb4c24270f7e842c8ddb1aad3120b546734576b53a52edf01296e4cec451dc"} Oct 30 04:06:58.948915 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.948846 4442 scope.go:122] "RemoveContainer" containerID="515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6" Oct 30 04:06:58.949258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.949237 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc" Oct 30 04:06:58.971241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.971080 4442 scope.go:122] "RemoveContainer" containerID="515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6" Oct 30 04:06:58.976142 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:06:58.973182 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6\": container with ID starting with 515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6 not found: ID does not exist" containerID="515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6" Oct 30 04:06:58.976142 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.973222 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6"} err="failed to get container status \"515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6\": rpc error: code = NotFound desc = could not find container \"515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6\": container with ID starting with 515ed08906a169e2b55aa6f1996164eec43eb8af57078ba60208af0a3521e6b6 not found: ID does not exist" Oct 30 04:06:58.978008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.977968 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl"] Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.997911 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.997943 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clkq6\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-kube-api-access-clkq6\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.997965 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.997980 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.997993 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.998006 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.998024 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30726b9-6954-4842-8ce1-c7c20fc326f1-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.998063 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.000666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:58.998077 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30726b9-6954-4842-8ce1-c7c20fc326f1-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:06:59.080911 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.080824 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc"] Oct 30 04:06:59.084259 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.084233 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-547a78-openshift-default-67dbdf45ff-4qjdc"] Oct 30 04:06:59.099362 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.099316 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:59.274996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.274963 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:59.297640 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.297609 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59"] Oct 30 04:06:59.412338 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.407611 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:59.506932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.506903 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:59.951003 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:06:59.950967 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:00.046270 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:00.046234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:00.111004 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:00.110974 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" path="/var/lib/kubelet/pods/d30726b9-6954-4842-8ce1-c7c20fc326f1/volumes" Oct 30 04:07:00.157442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:00.157409 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:00.256394 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:00.256361 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:00.583069 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:00.582950 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:00.615546 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:00.615511 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" podUID="2c775e5d-3134-40d3-8d81-4748781896a5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:01.394163 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:01.394129 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:01.841419 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:01.841379 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:03.074487 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.074447 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" podUID="bd6517be-64ee-4ab8-97e0-26b4d01784f6" containerName="main" containerID="cri-o://1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e" gracePeriod=45 Oct 30 04:07:03.365964 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.365938 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:07:03.404008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.403972 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-dshm\") pod \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " Oct 30 04:07:03.404219 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.404201 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6517be-64ee-4ab8-97e0-26b4d01784f6-kube-api-access-9h2n7\" (UniqueName: \"kubernetes.io/projected/bd6517be-64ee-4ab8-97e0-26b4d01784f6-kube-api-access-9h2n7\") pod \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " Oct 30 04:07:03.404296 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.404235 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-home\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-home\") pod \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " Oct 30 04:07:03.404296 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.404268 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-model-cache\") pod \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " Oct 30 04:07:03.404408 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.404315 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tmp-dir\") pod \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " Oct 30 04:07:03.404408 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.404349 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tls-certs\") pod \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\" (UID: \"bd6517be-64ee-4ab8-97e0-26b4d01784f6\") " Oct 30 04:07:03.405581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.405540 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-model-cache" pod "bd6517be-64ee-4ab8-97e0-26b4d01784f6" (UID: "bd6517be-64ee-4ab8-97e0-26b4d01784f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:03.405581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.405552 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tmp-dir" pod "bd6517be-64ee-4ab8-97e0-26b4d01784f6" (UID: "bd6517be-64ee-4ab8-97e0-26b4d01784f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:03.406325 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.406274 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-home" pod "bd6517be-64ee-4ab8-97e0-26b4d01784f6" (UID: "bd6517be-64ee-4ab8-97e0-26b4d01784f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:03.407910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.407884 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tls-certs" pod "bd6517be-64ee-4ab8-97e0-26b4d01784f6" (UID: "bd6517be-64ee-4ab8-97e0-26b4d01784f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:03.408045 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.408022 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-dshm" pod "bd6517be-64ee-4ab8-97e0-26b4d01784f6" (UID: "bd6517be-64ee-4ab8-97e0-26b4d01784f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:03.411204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.411176 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6517be-64ee-4ab8-97e0-26b4d01784f6-kube-api-access-9h2n7" pod "bd6517be-64ee-4ab8-97e0-26b4d01784f6" (UID: "bd6517be-64ee-4ab8-97e0-26b4d01784f6"). InnerVolumeSpecName "kube-api-access-9h2n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:03.505686 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.505637 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:03.505911 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.505891 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:03.505965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.505911 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h2n7\" (UniqueName: \"kubernetes.io/projected/bd6517be-64ee-4ab8-97e0-26b4d01784f6-kube-api-access-9h2n7\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:03.505965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.505926 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:03.505965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.505938 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:03.505965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.505946 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:03.505965 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.505953 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd6517be-64ee-4ab8-97e0-26b4d01784f6-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:03.985289 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.985253 4442 generic.go:356] "Generic (PLEG): container finished" podID="bd6517be-64ee-4ab8-97e0-26b4d01784f6" containerID="1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e" exitCode=0 Oct 30 04:07:03.985507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.985296 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" Oct 30 04:07:03.985507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.985334 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" event={"ID":"bd6517be-64ee-4ab8-97e0-26b4d01784f6","Type":"ContainerDied","Data":"1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e"} Oct 30 04:07:03.985507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.985381 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b" event={"ID":"bd6517be-64ee-4ab8-97e0-26b4d01784f6","Type":"ContainerDied","Data":"3adaf4e8b8b1657e7cd0a276b04b0e6a21670872ac304a93bf0befd6ee21c95a"} Oct 30 04:07:03.985507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.985403 4442 scope.go:122] "RemoveContainer" containerID="1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e" Oct 30 04:07:03.996140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.996125 4442 scope.go:122] "RemoveContainer" containerID="1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e" Oct 30 04:07:03.996463 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:03.996446 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e\": container with ID starting with 1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e not found: ID does not exist" containerID="1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e" Oct 30 04:07:03.996530 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:03.996471 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e"} err="failed to get container status \"1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e\": rpc error: code = NotFound desc = could not find container \"1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e\": container with ID starting with 1d02267be98198481c0e76d20cd0b3ff092bf884154b4f76d6721e0faaf9df7e not found: ID does not exist" Oct 30 04:07:04.036227 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:04.036196 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b"] Oct 30 04:07:04.047604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:04.047578 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-7d99a4/rate-test-model-7d99a4-kserve-5b79fd5d9-9t66b"] Oct 30 04:07:04.108201 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:04.108169 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6517be-64ee-4ab8-97e0-26b4d01784f6" path="/var/lib/kubelet/pods/bd6517be-64ee-4ab8-97e0-26b4d01784f6/volumes" Oct 30 04:07:04.299157 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:04.299108 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" podUID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" containerName="payload-pre-processing" containerID="cri-o://12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571" gracePeriod=25 Oct 30 04:07:05.055712 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.055685 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:05.578543 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.578484 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 04:07:05.640730 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.640696 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c775e5d-3134-40d3-8d81-4748781896a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c775e5d-3134-40d3-8d81-4748781896a5-plugins-config-volume\") pod \"2c775e5d-3134-40d3-8d81-4748781896a5\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " Oct 30 04:07:05.640993 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.640976 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c775e5d-3134-40d3-8d81-4748781896a5-kube-api-access-m84kr\" (UniqueName: \"kubernetes.io/projected/2c775e5d-3134-40d3-8d81-4748781896a5-kube-api-access-m84kr\") pod \"2c775e5d-3134-40d3-8d81-4748781896a5\" (UID: \"2c775e5d-3134-40d3-8d81-4748781896a5\") " Oct 30 04:07:05.641186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.641158 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c775e5d-3134-40d3-8d81-4748781896a5-plugins-config-volume" pod "2c775e5d-3134-40d3-8d81-4748781896a5" (UID: "2c775e5d-3134-40d3-8d81-4748781896a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:05.641474 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.641452 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c775e5d-3134-40d3-8d81-4748781896a5-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:05.643276 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.643248 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c775e5d-3134-40d3-8d81-4748781896a5-kube-api-access-m84kr" pod "2c775e5d-3134-40d3-8d81-4748781896a5" (UID: "2c775e5d-3134-40d3-8d81-4748781896a5"). InnerVolumeSpecName "kube-api-access-m84kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:05.742574 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:05.742538 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84kr\" (UniqueName: \"kubernetes.io/projected/2c775e5d-3134-40d3-8d81-4748781896a5-kube-api-access-m84kr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:06.004258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.004222 4442 generic.go:356] "Generic (PLEG): container finished" podID="2c775e5d-3134-40d3-8d81-4748781896a5" containerID="8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072" exitCode=0 Oct 30 04:07:06.004518 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.004280 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" Oct 30 04:07:06.004518 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.004346 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" event={"ID":"2c775e5d-3134-40d3-8d81-4748781896a5","Type":"ContainerDied","Data":"8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072"} Oct 30 04:07:06.004518 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.004425 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml" event={"ID":"2c775e5d-3134-40d3-8d81-4748781896a5","Type":"ContainerDied","Data":"667f2b59cd7ea21cca19cf4cdcb09f91f714940162e7b006840c0610c828736b"} Oct 30 04:07:06.004518 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.004455 4442 scope.go:122] "RemoveContainer" containerID="8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072" Oct 30 04:07:06.020831 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.020809 4442 scope.go:122] "RemoveContainer" containerID="8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072" Oct 30 04:07:06.021165 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:06.021137 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072\": container with ID starting with 8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072 not found: ID does not exist" containerID="8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072" Oct 30 04:07:06.021234 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.021174 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072"} err="failed to get container status \"8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072\": rpc error: code = NotFound desc = could not find container \"8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072\": container with ID starting with 8b1c3dc4a5305df4c44745a164838b8aed7ae9d27fa70b0429f317cd1173f072 not found: ID does not exist" Oct 30 04:07:06.055352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.055317 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml"] Oct 30 04:07:06.065682 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.065662 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-817531-5f8bb9b4c6-vzrml"] Oct 30 04:07:06.108123 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.108097 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c775e5d-3134-40d3-8d81-4748781896a5" path="/var/lib/kubelet/pods/2c775e5d-3134-40d3-8d81-4748781896a5/volumes" Oct 30 04:07:06.407524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.407445 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:06.983827 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.983789 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" podUID="452ef5f7-56c5-4b2b-863c-58414354fca5" containerName="main" containerID="cri-o://e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e" gracePeriod=45 Oct 30 04:07:06.988932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:06.988875 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" podUID="452ef5f7-56c5-4b2b-863c-58414354fca5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.233:8000/ready\": EOF" Oct 30 04:07:07.081240 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.081214 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:07.275677 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.275608 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:07.344494 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.344460 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" podUID="2b37ede5-6f78-44aa-9211-4e23d05941c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:07.373822 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.373731 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:07.411800 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.411777 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:07:07.466189 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.466160 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:07.494948 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.494918 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452ef5f7-56c5-4b2b-863c-58414354fca5-kube-api-access-4fpk8\" (UniqueName: \"kubernetes.io/projected/452ef5f7-56c5-4b2b-863c-58414354fca5-kube-api-access-4fpk8\") pod \"452ef5f7-56c5-4b2b-863c-58414354fca5\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " Oct 30 04:07:07.495140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.494967 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-model-cache\") pod \"452ef5f7-56c5-4b2b-863c-58414354fca5\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " Oct 30 04:07:07.495140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495073 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-tmp-dir\") pod \"452ef5f7-56c5-4b2b-863c-58414354fca5\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " Oct 30 04:07:07.495140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495104 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-home\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-home\") pod \"452ef5f7-56c5-4b2b-863c-58414354fca5\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " Oct 30 04:07:07.495329 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495146 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/452ef5f7-56c5-4b2b-863c-58414354fca5-tls-certs\" (UniqueName: \"kubernetes.io/secret/452ef5f7-56c5-4b2b-863c-58414354fca5-tls-certs\") pod \"452ef5f7-56c5-4b2b-863c-58414354fca5\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " Oct 30 04:07:07.495329 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495189 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-dshm\") pod \"452ef5f7-56c5-4b2b-863c-58414354fca5\" (UID: \"452ef5f7-56c5-4b2b-863c-58414354fca5\") " Oct 30 04:07:07.495438 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495323 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-tmp-dir" pod "452ef5f7-56c5-4b2b-863c-58414354fca5" (UID: "452ef5f7-56c5-4b2b-863c-58414354fca5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:07.495438 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495371 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-model-cache" pod "452ef5f7-56c5-4b2b-863c-58414354fca5" (UID: "452ef5f7-56c5-4b2b-863c-58414354fca5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:07.495438 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495400 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-home" pod "452ef5f7-56c5-4b2b-863c-58414354fca5" (UID: "452ef5f7-56c5-4b2b-863c-58414354fca5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:07.495721 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495700 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:07.495721 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495720 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:07.496065 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.495728 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:07.497451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.497426 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-dshm" pod "452ef5f7-56c5-4b2b-863c-58414354fca5" (UID: "452ef5f7-56c5-4b2b-863c-58414354fca5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:07.497565 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.497510 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452ef5f7-56c5-4b2b-863c-58414354fca5-kube-api-access-4fpk8" pod "452ef5f7-56c5-4b2b-863c-58414354fca5" (UID: "452ef5f7-56c5-4b2b-863c-58414354fca5"). InnerVolumeSpecName "kube-api-access-4fpk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:07.497627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.497584 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452ef5f7-56c5-4b2b-863c-58414354fca5-tls-certs" pod "452ef5f7-56c5-4b2b-863c-58414354fca5" (UID: "452ef5f7-56c5-4b2b-863c-58414354fca5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:07.564675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.564448 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:07.597983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.597891 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpk8\" (UniqueName: \"kubernetes.io/projected/452ef5f7-56c5-4b2b-863c-58414354fca5-kube-api-access-4fpk8\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:07.597983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.597939 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ef5f7-56c5-4b2b-863c-58414354fca5-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:07.597983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:07.597954 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/452ef5f7-56c5-4b2b-863c-58414354fca5-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:08.020013 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.019983 4442 generic.go:356] "Generic (PLEG): container finished" podID="452ef5f7-56c5-4b2b-863c-58414354fca5" containerID="e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e" exitCode=0 Oct 30 04:07:08.020491 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.020025 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" Oct 30 04:07:08.020491 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.020058 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" event={"ID":"452ef5f7-56c5-4b2b-863c-58414354fca5","Type":"ContainerDied","Data":"e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e"} Oct 30 04:07:08.020491 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.020101 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw" event={"ID":"452ef5f7-56c5-4b2b-863c-58414354fca5","Type":"ContainerDied","Data":"6e736e2d0dc4c05d5e23a59525470621ad07ba0ff28a97e9a867e928ac28817d"} Oct 30 04:07:08.020491 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.020125 4442 scope.go:122] "RemoveContainer" containerID="e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e" Oct 30 04:07:08.033108 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.033091 4442 scope.go:122] "RemoveContainer" containerID="e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e" Oct 30 04:07:08.033397 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:08.033369 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e\": container with ID starting with e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e not found: ID does not exist" containerID="e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e" Oct 30 04:07:08.033513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.033408 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e"} err="failed to get container status \"e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e\": rpc error: code = NotFound desc = could not find container \"e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e\": container with ID starting with e1f7e583dee717710fc035125f737183c9914efb9953c9ce3c883c5ca6ecf35e not found: ID does not exist" Oct 30 04:07:08.070139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.070111 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw"] Oct 30 04:07:08.080791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.080743 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-852107/rate-test-model-852107-kserve-7c896c87d9-mwzlw"] Oct 30 04:07:08.081391 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.081370 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:08.109680 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.109651 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452ef5f7-56c5-4b2b-863c-58414354fca5" path="/var/lib/kubelet/pods/452ef5f7-56c5-4b2b-863c-58414354fca5/volumes" Oct 30 04:07:08.126700 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.126676 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr"] Oct 30 04:07:08.164587 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.164557 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9"] Oct 30 04:07:08.284842 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.284700 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:08.391780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.388147 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:08.496856 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.496796 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:08.818010 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:08.817977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:09.055336 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.055304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:09.160705 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.160616 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:09.360982 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.360954 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:09.800591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.800544 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 04:07:09.940040 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.940010 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b37ede5-6f78-44aa-9211-4e23d05941c2-kube-api-access-psvfs\" (UniqueName: \"kubernetes.io/projected/2b37ede5-6f78-44aa-9211-4e23d05941c2-kube-api-access-psvfs\") pod \"2b37ede5-6f78-44aa-9211-4e23d05941c2\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " Oct 30 04:07:09.940316 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.940294 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b37ede5-6f78-44aa-9211-4e23d05941c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b37ede5-6f78-44aa-9211-4e23d05941c2-plugins-config-volume\") pod \"2b37ede5-6f78-44aa-9211-4e23d05941c2\" (UID: \"2b37ede5-6f78-44aa-9211-4e23d05941c2\") " Oct 30 04:07:09.940658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.940631 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b37ede5-6f78-44aa-9211-4e23d05941c2-plugins-config-volume" pod "2b37ede5-6f78-44aa-9211-4e23d05941c2" (UID: "2b37ede5-6f78-44aa-9211-4e23d05941c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:09.940986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.940824 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b37ede5-6f78-44aa-9211-4e23d05941c2-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:09.942362 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:09.942342 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b37ede5-6f78-44aa-9211-4e23d05941c2-kube-api-access-psvfs" pod "2b37ede5-6f78-44aa-9211-4e23d05941c2" (UID: "2b37ede5-6f78-44aa-9211-4e23d05941c2"). InnerVolumeSpecName "kube-api-access-psvfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:10.018148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.018115 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:10.040363 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.040331 4442 generic.go:356] "Generic (PLEG): container finished" podID="2b37ede5-6f78-44aa-9211-4e23d05941c2" containerID="993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7" exitCode=0 Oct 30 04:07:10.040545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.040389 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" Oct 30 04:07:10.040545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.040406 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" event={"ID":"2b37ede5-6f78-44aa-9211-4e23d05941c2","Type":"ContainerDied","Data":"993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7"} Oct 30 04:07:10.040545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.040503 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv" event={"ID":"2b37ede5-6f78-44aa-9211-4e23d05941c2","Type":"ContainerDied","Data":"9ca5b7042ff9934a9c179549926b7581128cb179a55f94902a63450c14c2d5ca"} Oct 30 04:07:10.040545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.040542 4442 scope.go:122] "RemoveContainer" containerID="993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7" Oct 30 04:07:10.041911 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.041848 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvfs\" (UniqueName: \"kubernetes.io/projected/2b37ede5-6f78-44aa-9211-4e23d05941c2-kube-api-access-psvfs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:10.051177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.051160 4442 scope.go:122] "RemoveContainer" containerID="993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7" Oct 30 04:07:10.051499 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:10.051465 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7\": container with ID starting with 993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7 not found: ID does not exist" containerID="993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7" Oct 30 04:07:10.051573 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.051514 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7"} err="failed to get container status \"993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7\": rpc error: code = NotFound desc = could not find container \"993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7\": container with ID starting with 993da287f6ab02f1a3cc9f181863fb61a3151665ceade651da799da1bd0fc0a7 not found: ID does not exist" Oct 30 04:07:10.088098 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.088072 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv"] Oct 30 04:07:10.091495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.091477 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-547a78-585cbc8c7b-t5sbv"] Oct 30 04:07:10.107413 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.107382 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b37ede5-6f78-44aa-9211-4e23d05941c2" path="/var/lib/kubelet/pods/2b37ede5-6f78-44aa-9211-4e23d05941c2/volumes" Oct 30 04:07:10.510061 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:10.510019 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:12.217711 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:12.217677 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:12.316540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:12.316504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:12.688108 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:12.688070 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" podUID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:12.956549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:12.956464 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:13.127182 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:13.127144 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" podUID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" containerName="payload-processing" containerID="cri-o://a3c17e29e03104c1974eb7620f5751118fa3925131eec74a00f8652ad8c309f5" gracePeriod=25 Oct 30 04:07:13.165614 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:13.165577 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" podUID="fd668d56-aefe-46e4-aa96-6d02036bac6f" containerName="payload-pre-processing" containerID="cri-o://d140c6dbf32eede2b07b23748914d78402a5cd4031abc844ac49c0048c95a0d0" gracePeriod=25 Oct 30 04:07:13.322782 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:13.322733 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:13.828261 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:13.828224 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:13.970482 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:13.970443 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:14.041144 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.041099 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" podUID="7b752eca-931a-4222-8569-8bae91568839" containerName="main" containerID="cri-o://fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9" gracePeriod=45 Oct 30 04:07:14.070600 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.070567 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:14.319296 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.319271 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 04:07:14.409101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409071 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-home\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-home\") pod \"7b752eca-931a-4222-8569-8bae91568839\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " Oct 30 04:07:14.409591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409134 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b752eca-931a-4222-8569-8bae91568839-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b752eca-931a-4222-8569-8bae91568839-tls-certs\") pod \"7b752eca-931a-4222-8569-8bae91568839\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " Oct 30 04:07:14.409591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409175 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-model-cache\") pod \"7b752eca-931a-4222-8569-8bae91568839\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " Oct 30 04:07:14.409591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409331 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-dshm\") pod \"7b752eca-931a-4222-8569-8bae91568839\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " Oct 30 04:07:14.409591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409463 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-model-cache" pod "7b752eca-931a-4222-8569-8bae91568839" (UID: "7b752eca-931a-4222-8569-8bae91568839"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:14.409591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409494 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b752eca-931a-4222-8569-8bae91568839-kube-api-access-fht67\" (UniqueName: \"kubernetes.io/projected/7b752eca-931a-4222-8569-8bae91568839-kube-api-access-fht67\") pod \"7b752eca-931a-4222-8569-8bae91568839\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " Oct 30 04:07:14.409591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409532 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-tmp-dir\") pod \"7b752eca-931a-4222-8569-8bae91568839\" (UID: \"7b752eca-931a-4222-8569-8bae91568839\") " Oct 30 04:07:14.409956 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.409784 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-home" pod "7b752eca-931a-4222-8569-8bae91568839" (UID: "7b752eca-931a-4222-8569-8bae91568839"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:14.410060 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.410036 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-tmp-dir" pod "7b752eca-931a-4222-8569-8bae91568839" (UID: "7b752eca-931a-4222-8569-8bae91568839"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:14.410488 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.410465 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:14.410620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.410492 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:14.410620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.410505 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:14.412696 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.412637 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-dshm" pod "7b752eca-931a-4222-8569-8bae91568839" (UID: "7b752eca-931a-4222-8569-8bae91568839"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:14.412881 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.412861 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b752eca-931a-4222-8569-8bae91568839-tls-certs" pod "7b752eca-931a-4222-8569-8bae91568839" (UID: "7b752eca-931a-4222-8569-8bae91568839"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:14.414242 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.414215 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b752eca-931a-4222-8569-8bae91568839-kube-api-access-fht67" pod "7b752eca-931a-4222-8569-8bae91568839" (UID: "7b752eca-931a-4222-8569-8bae91568839"). InnerVolumeSpecName "kube-api-access-fht67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:14.511713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.511672 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fht67\" (UniqueName: \"kubernetes.io/projected/7b752eca-931a-4222-8569-8bae91568839-kube-api-access-fht67\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:14.511713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.511699 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b752eca-931a-4222-8569-8bae91568839-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:14.511713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.511711 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b752eca-931a-4222-8569-8bae91568839-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:14.762803 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.762764 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2"] Oct 30 04:07:14.798275 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.798242 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj"] Oct 30 04:07:14.902864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:14.902835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:15.002236 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.002178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:15.077503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.077408 4442 generic.go:356] "Generic (PLEG): container finished" podID="7b752eca-931a-4222-8569-8bae91568839" containerID="fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9" exitCode=0 Oct 30 04:07:15.077503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.077459 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" Oct 30 04:07:15.077503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.077489 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" event={"ID":"7b752eca-931a-4222-8569-8bae91568839","Type":"ContainerDied","Data":"fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9"} Oct 30 04:07:15.077727 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.077534 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl" event={"ID":"7b752eca-931a-4222-8569-8bae91568839","Type":"ContainerDied","Data":"3a02434c2863553b1c2858d5afaf803ea8430fdae40a1a8c571882ea46c0aa7c"} Oct 30 04:07:15.077727 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.077551 4442 scope.go:122] "RemoveContainer" containerID="fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9" Oct 30 04:07:15.089664 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.089647 4442 scope.go:122] "RemoveContainer" containerID="fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9" Oct 30 04:07:15.089967 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:15.089946 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9\": container with ID starting with fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9 not found: ID does not exist" containerID="fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9" Oct 30 04:07:15.090056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.089976 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9"} err="failed to get container status \"fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9\": rpc error: code = NotFound desc = could not find container \"fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9\": container with ID starting with fb5b09a63babf6893f353311e63a7581610ce3f37686974662b3349df830e8d9 not found: ID does not exist" Oct 30 04:07:15.128236 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.128205 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl"] Oct 30 04:07:15.134850 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.134822 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-e44313/distinct-w1-e44313-kserve-667854f86d-sz2vl"] Oct 30 04:07:15.141203 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.141172 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" podUID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:15.839408 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:15.839375 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:16.107475 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:16.107390 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b752eca-931a-4222-8569-8bae91568839" path="/var/lib/kubelet/pods/7b752eca-931a-4222-8569-8bae91568839/volumes" Oct 30 04:07:16.461070 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:16.460941 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:16.957893 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:16.957847 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:17.152970 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:17.152937 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" podUID="fd668d56-aefe-46e4-aa96-6d02036bac6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:17.403888 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:17.403856 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:18.738542 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:18.738509 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42"] Oct 30 04:07:18.845359 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:18.845326 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:19.103973 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:19.100352 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:19.763697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:19.763663 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" podUID="60c7cd60-845b-4415-a560-1e913027eeb6" containerName="payload-processing" containerID="cri-o://c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723" gracePeriod=25 Oct 30 04:07:19.799390 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:19.799346 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" podUID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" containerName="payload-pre-processing" containerID="cri-o://598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4" gracePeriod=25 Oct 30 04:07:19.830511 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:19.830470 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:20.230648 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.230563 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" podUID="60c7cd60-845b-4415-a560-1e913027eeb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:20.343530 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.343496 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr"] Oct 30 04:07:20.343764 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.343714 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" podUID="15707df2-3b1c-43a8-8a99-b1f10231cb00" containerName="maas-api" containerID="cri-o://d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5" gracePeriod=30 Oct 30 04:07:20.399587 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.399556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:20.679318 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.679289 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 04:07:20.774254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.771512 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15707df2-3b1c-43a8-8a99-b1f10231cb00-kube-api-access-5wfbd\" (UniqueName: \"kubernetes.io/projected/15707df2-3b1c-43a8-8a99-b1f10231cb00-kube-api-access-5wfbd\") pod \"15707df2-3b1c-43a8-8a99-b1f10231cb00\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " Oct 30 04:07:20.774254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.771649 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15707df2-3b1c-43a8-8a99-b1f10231cb00-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15707df2-3b1c-43a8-8a99-b1f10231cb00-openshift-service-ca\") pod \"15707df2-3b1c-43a8-8a99-b1f10231cb00\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " Oct 30 04:07:20.774254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.771801 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-metrics-certs\") pod \"15707df2-3b1c-43a8-8a99-b1f10231cb00\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " Oct 30 04:07:20.774254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.771886 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-maas-api-tls\") pod \"15707df2-3b1c-43a8-8a99-b1f10231cb00\" (UID: \"15707df2-3b1c-43a8-8a99-b1f10231cb00\") " Oct 30 04:07:20.777500 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.777455 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15707df2-3b1c-43a8-8a99-b1f10231cb00-openshift-service-ca" pod "15707df2-3b1c-43a8-8a99-b1f10231cb00" (UID: "15707df2-3b1c-43a8-8a99-b1f10231cb00"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:20.780212 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.780171 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15707df2-3b1c-43a8-8a99-b1f10231cb00-kube-api-access-5wfbd" pod "15707df2-3b1c-43a8-8a99-b1f10231cb00" (UID: "15707df2-3b1c-43a8-8a99-b1f10231cb00"). InnerVolumeSpecName "kube-api-access-5wfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:20.788320 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.787855 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-maas-api-tls" pod "15707df2-3b1c-43a8-8a99-b1f10231cb00" (UID: "15707df2-3b1c-43a8-8a99-b1f10231cb00"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:20.790219 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.790180 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-metrics-certs" pod "15707df2-3b1c-43a8-8a99-b1f10231cb00" (UID: "15707df2-3b1c-43a8-8a99-b1f10231cb00"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:20.873833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.873795 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wfbd\" (UniqueName: \"kubernetes.io/projected/15707df2-3b1c-43a8-8a99-b1f10231cb00-kube-api-access-5wfbd\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:20.873833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.873828 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15707df2-3b1c-43a8-8a99-b1f10231cb00-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:20.873833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.873839 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:20.874175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:20.873848 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15707df2-3b1c-43a8-8a99-b1f10231cb00-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:21.119135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.119052 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:21.135999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.135968 4442 generic.go:356] "Generic (PLEG): container finished" podID="15707df2-3b1c-43a8-8a99-b1f10231cb00" containerID="d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5" exitCode=0 Oct 30 04:07:21.136229 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.136013 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" Oct 30 04:07:21.136229 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.136064 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" event={"ID":"15707df2-3b1c-43a8-8a99-b1f10231cb00","Type":"ContainerDied","Data":"d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5"} Oct 30 04:07:21.136229 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.136125 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr" event={"ID":"15707df2-3b1c-43a8-8a99-b1f10231cb00","Type":"ContainerDied","Data":"5bce4891c47b0b826d7cb034e2afbd69fba45588ec94d6e5dffcae4c5316984e"} Oct 30 04:07:21.136229 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.136159 4442 scope.go:122] "RemoveContainer" containerID="d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5" Oct 30 04:07:21.147851 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.147833 4442 scope.go:122] "RemoveContainer" containerID="d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5" Oct 30 04:07:21.148134 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:21.148115 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5\": container with ID starting with d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5 not found: ID does not exist" containerID="d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5" Oct 30 04:07:21.148228 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.148141 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5"} err="failed to get container status \"d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5\": rpc error: code = NotFound desc = could not find container \"d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5\": container with ID starting with d074e2e2aac458fcdcb4de53bc8887d3952dedf1715643a77da5d08b4a3bf7e5 not found: ID does not exist" Oct 30 04:07:21.213926 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.213845 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr"] Oct 30 04:07:21.216617 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.216584 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-da4553-54df678dbb-l2mdr"] Oct 30 04:07:21.651042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:21.651011 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:22.107745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:22.107712 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15707df2-3b1c-43a8-8a99-b1f10231cb00" path="/var/lib/kubelet/pods/15707df2-3b1c-43a8-8a99-b1f10231cb00/volumes" Oct 30 04:07:22.394795 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:22.394694 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:22.688174 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:22.688089 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" podUID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:22.734902 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:22.734868 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:23.330546 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:23.330509 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:23.447587 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:23.447554 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:23.547776 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:23.547718 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:24.171183 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.171150 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:24.295067 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.295036 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:24.523493 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.523462 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs"] Oct 30 04:07:24.524434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.523713 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" containerID="cri-o://b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db" gracePeriod=30 Oct 30 04:07:24.568004 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.567974 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:24.588917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.588896 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 04:07:24.618962 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.618937 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-kube-api-access-pmhfj\" (UniqueName: \"kubernetes.io/projected/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-kube-api-access-pmhfj\") pod \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " Oct 30 04:07:24.619127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.619112 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-plugins-config-volume\") pod \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\" (UID: \"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14\") " Oct 30 04:07:24.619368 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.619347 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-plugins-config-volume" pod "25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" (UID: "25b33eb0-d5c0-4160-8a3f-3c53d63f5a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:24.619547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.619532 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:24.621261 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.621235 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-kube-api-access-pmhfj" pod "25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" (UID: "25b33eb0-d5c0-4160-8a3f-3c53d63f5a14"). InnerVolumeSpecName "kube-api-access-pmhfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:24.732950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:24.732814 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhfj\" (UniqueName: \"kubernetes.io/projected/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14-kube-api-access-pmhfj\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:24.861616 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:24.861581 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:07:25.140719 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.140682 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" podUID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:25.163107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.163078 4442 generic.go:356] "Generic (PLEG): container finished" podID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" containerID="12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571" exitCode=0 Oct 30 04:07:25.163256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.163140 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" Oct 30 04:07:25.163256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.163161 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" event={"ID":"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14","Type":"ContainerDied","Data":"12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571"} Oct 30 04:07:25.163256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.163221 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59" event={"ID":"25b33eb0-d5c0-4160-8a3f-3c53d63f5a14","Type":"ContainerDied","Data":"816dabf3fc741cd4d5f819089b722f8941598b099bc867cf30c76e3ad5c3e2a4"} Oct 30 04:07:25.163256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.163245 4442 scope.go:122] "RemoveContainer" containerID="12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571" Oct 30 04:07:25.173561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.173545 4442 scope.go:122] "RemoveContainer" containerID="12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571" Oct 30 04:07:25.173853 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:25.173832 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571\": container with ID starting with 12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571 not found: ID does not exist" containerID="12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571" Oct 30 04:07:25.173974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.173861 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571"} err="failed to get container status \"12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571\": rpc error: code = NotFound desc = could not find container \"12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571\": container with ID starting with 12934fbc86eb83d6151423667f6f32eb05067bcf57aa9b86b8c1fc95cc961571 not found: ID does not exist" Oct 30 04:07:25.215506 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.215478 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59"] Oct 30 04:07:25.218122 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.218101 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-e10c0f-cbd8676c5-vbs59"] Oct 30 04:07:25.393449 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.393364 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:25.495766 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.495708 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:25.595127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.595100 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:25.887746 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.887712 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6"] Oct 30 04:07:25.888014 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.887974 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" podUID="38351d05-2730-4456-8bb6-46226286b4c1" containerName="maas-api" containerID="cri-o://7ccba4f0e541cdde905d323f2675d1d857991e2409ade61ca7c87f05844d7899" gracePeriod=30 Oct 30 04:07:25.943061 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:25.943029 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:26.110806 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.110774 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" path="/var/lib/kubelet/pods/25b33eb0-d5c0-4160-8a3f-3c53d63f5a14/volumes" Oct 30 04:07:26.250883 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.238017 4442 generic.go:356] "Generic (PLEG): container finished" podID="38351d05-2730-4456-8bb6-46226286b4c1" containerID="7ccba4f0e541cdde905d323f2675d1d857991e2409ade61ca7c87f05844d7899" exitCode=0 Oct 30 04:07:26.250883 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.238220 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" event={"ID":"38351d05-2730-4456-8bb6-46226286b4c1","Type":"ContainerDied","Data":"7ccba4f0e541cdde905d323f2675d1d857991e2409ade61ca7c87f05844d7899"} Oct 30 04:07:26.332995 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.332953 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 04:07:26.399988 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.399953 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38351d05-2730-4456-8bb6-46226286b4c1-kube-api-access-7hzs4\" (UniqueName: \"kubernetes.io/projected/38351d05-2730-4456-8bb6-46226286b4c1-kube-api-access-7hzs4\") pod \"38351d05-2730-4456-8bb6-46226286b4c1\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " Oct 30 04:07:26.400195 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.400107 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-metrics-certs\") pod \"38351d05-2730-4456-8bb6-46226286b4c1\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " Oct 30 04:07:26.400195 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.400178 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38351d05-2730-4456-8bb6-46226286b4c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38351d05-2730-4456-8bb6-46226286b4c1-openshift-service-ca\") pod \"38351d05-2730-4456-8bb6-46226286b4c1\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " Oct 30 04:07:26.400316 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.400294 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls\") pod \"38351d05-2730-4456-8bb6-46226286b4c1\" (UID: \"38351d05-2730-4456-8bb6-46226286b4c1\") " Oct 30 04:07:26.401363 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.401336 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38351d05-2730-4456-8bb6-46226286b4c1-openshift-service-ca" pod "38351d05-2730-4456-8bb6-46226286b4c1" (UID: "38351d05-2730-4456-8bb6-46226286b4c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:26.403136 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.403110 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38351d05-2730-4456-8bb6-46226286b4c1-kube-api-access-7hzs4" pod "38351d05-2730-4456-8bb6-46226286b4c1" (UID: "38351d05-2730-4456-8bb6-46226286b4c1"). InnerVolumeSpecName "kube-api-access-7hzs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:26.403277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.403259 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-metrics-certs" pod "38351d05-2730-4456-8bb6-46226286b4c1" (UID: "38351d05-2730-4456-8bb6-46226286b4c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:26.403621 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.403593 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls" pod "38351d05-2730-4456-8bb6-46226286b4c1" (UID: "38351d05-2730-4456-8bb6-46226286b4c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:26.501940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.501840 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:26.501940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.501879 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38351d05-2730-4456-8bb6-46226286b4c1-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:26.501940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.501893 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38351d05-2730-4456-8bb6-46226286b4c1-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:26.501940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.501902 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hzs4\" (UniqueName: \"kubernetes.io/projected/38351d05-2730-4456-8bb6-46226286b4c1-kube-api-access-7hzs4\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:26.762249 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:26.762176 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:27.153384 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:27.153300 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" podUID="fd668d56-aefe-46e4-aa96-6d02036bac6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:27.245958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:27.245927 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" Oct 30 04:07:27.246152 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:27.245960 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6" event={"ID":"38351d05-2730-4456-8bb6-46226286b4c1","Type":"ContainerDied","Data":"4d1ade95c9e449333acb5b16c3d447d0c2e1e3bc158b09d271aa246dc53bd2f3"} Oct 30 04:07:27.246152 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:27.246020 4442 scope.go:122] "RemoveContainer" containerID="7ccba4f0e541cdde905d323f2675d1d857991e2409ade61ca7c87f05844d7899" Oct 30 04:07:27.287514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:27.287484 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:27.293527 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:27.293502 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6"] Oct 30 04:07:27.304383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:27.304361 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-e44313-587c745445-jg7g6"] Oct 30 04:07:28.107472 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:28.107434 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38351d05-2730-4456-8bb6-46226286b4c1" path="/var/lib/kubelet/pods/38351d05-2730-4456-8bb6-46226286b4c1/volumes" Oct 30 04:07:28.110208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:28.110186 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:28.220595 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:28.220556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:28.395939 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:28.395830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:28.800062 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:28.800036 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:29.136277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.136185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:29.224706 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.224670 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" podUID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:29.789888 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.789788 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk"] Oct 30 04:07:29.790059 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.790008 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" podUID="d7867e74-3d84-4780-88b8-a78222af9a95" containerName="maas-api" containerID="cri-o://e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8" gracePeriod=30 Oct 30 04:07:29.862464 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.862434 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:29.887231 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.887202 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:07:29.957391 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957357 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-certs\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957483 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-credential-socket\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957514 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-envoy\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957545 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-data\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957635 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-token\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-token\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957674 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-podinfo\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957720 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istiod-ca-cert\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957821 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-kube-api-access-jr69d\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-kube-api-access-jr69d\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957880 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-socket\") pod \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\" (UID: \"e59a5ab2-179c-47fc-a7c3-4dae630f458d\") " Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.957628 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-certs" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.958270 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-socket" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.958425 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-data" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.958715 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-credential-socket" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.960691 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-token" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.960970 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istiod-ca-cert" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:29.967008 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.961884 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-podinfo" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:07:29.972289 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.972262 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-kube-api-access-jr69d" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "kube-api-access-jr69d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:29.972689 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:29.972662 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-envoy" pod "e59a5ab2-179c-47fc-a7c3-4dae630f458d" (UID: "e59a5ab2-179c-47fc-a7c3-4dae630f458d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.066952 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr69d\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-kube-api-access-jr69d\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.066986 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.066999 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.067012 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.067025 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.067038 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.067051 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.067062 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.067193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.067073 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e59a5ab2-179c-47fc-a7c3-4dae630f458d-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.111042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.111014 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 04:07:30.167930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.167896 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-metrics-certs\") pod \"d7867e74-3d84-4780-88b8-a78222af9a95\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " Oct 30 04:07:30.168432 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.167965 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-maas-api-tls\") pod \"d7867e74-3d84-4780-88b8-a78222af9a95\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " Oct 30 04:07:30.168432 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.167992 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7867e74-3d84-4780-88b8-a78222af9a95-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7867e74-3d84-4780-88b8-a78222af9a95-openshift-service-ca\") pod \"d7867e74-3d84-4780-88b8-a78222af9a95\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " Oct 30 04:07:30.168432 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.168215 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7867e74-3d84-4780-88b8-a78222af9a95-kube-api-access-trdtx\" (UniqueName: \"kubernetes.io/projected/d7867e74-3d84-4780-88b8-a78222af9a95-kube-api-access-trdtx\") pod \"d7867e74-3d84-4780-88b8-a78222af9a95\" (UID: \"d7867e74-3d84-4780-88b8-a78222af9a95\") " Oct 30 04:07:30.168635 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.168461 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7867e74-3d84-4780-88b8-a78222af9a95-openshift-service-ca" pod "d7867e74-3d84-4780-88b8-a78222af9a95" (UID: "d7867e74-3d84-4780-88b8-a78222af9a95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:30.169296 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.169273 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7867e74-3d84-4780-88b8-a78222af9a95-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.170650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.170622 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7867e74-3d84-4780-88b8-a78222af9a95-kube-api-access-trdtx" pod "d7867e74-3d84-4780-88b8-a78222af9a95" (UID: "d7867e74-3d84-4780-88b8-a78222af9a95"). InnerVolumeSpecName "kube-api-access-trdtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:30.170828 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.170800 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-metrics-certs" pod "d7867e74-3d84-4780-88b8-a78222af9a95" (UID: "d7867e74-3d84-4780-88b8-a78222af9a95"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:30.170918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.170833 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-maas-api-tls" pod "d7867e74-3d84-4780-88b8-a78222af9a95" (UID: "d7867e74-3d84-4780-88b8-a78222af9a95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:30.229929 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.229888 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" podUID="60c7cd60-845b-4415-a560-1e913027eeb6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:30.295864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.295734 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trdtx\" (UniqueName: \"kubernetes.io/projected/d7867e74-3d84-4780-88b8-a78222af9a95-kube-api-access-trdtx\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.295864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.295794 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.295864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.295808 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7867e74-3d84-4780-88b8-a78222af9a95-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:30.345166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.339685 4442 generic.go:356] "Generic (PLEG): container finished" podID="d7867e74-3d84-4780-88b8-a78222af9a95" containerID="e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8" exitCode=0 Oct 30 04:07:30.345166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.339866 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" event={"ID":"d7867e74-3d84-4780-88b8-a78222af9a95","Type":"ContainerDied","Data":"e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8"} Oct 30 04:07:30.345166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.339936 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" event={"ID":"d7867e74-3d84-4780-88b8-a78222af9a95","Type":"ContainerDied","Data":"70738a5417d708479c36deb52445e7959c8e6707c36440807a27995c67f064d4"} Oct 30 04:07:30.345166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.339967 4442 scope.go:122] "RemoveContainer" containerID="e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8" Oct 30 04:07:30.345166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.340183 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk" Oct 30 04:07:30.349305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.349267 4442 generic.go:356] "Generic (PLEG): container finished" podID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerID="b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db" exitCode=0 Oct 30 04:07:30.349473 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.349452 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" event={"ID":"e59a5ab2-179c-47fc-a7c3-4dae630f458d","Type":"ContainerDied","Data":"b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db"} Oct 30 04:07:30.349547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.349502 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" event={"ID":"e59a5ab2-179c-47fc-a7c3-4dae630f458d","Type":"ContainerDied","Data":"45e74ea7925f224291fbcd85753d6f428fd5aa13af9ee0662dd2e93301ff3459"} Oct 30 04:07:30.349609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.349586 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs" Oct 30 04:07:30.366949 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.366921 4442 scope.go:122] "RemoveContainer" containerID="e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8" Oct 30 04:07:30.367344 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:30.367319 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8\": container with ID starting with e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8 not found: ID does not exist" containerID="e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8" Oct 30 04:07:30.367468 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.367364 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8"} err="failed to get container status \"e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8\": rpc error: code = NotFound desc = could not find container \"e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8\": container with ID starting with e90bd62bc05a562122518e4cf01ffa5e409352d8a95501c505bf23acf84233a8 not found: ID does not exist" Oct 30 04:07:30.367468 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.367385 4442 scope.go:122] "RemoveContainer" containerID="b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db" Oct 30 04:07:30.377304 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.377285 4442 scope.go:122] "RemoveContainer" containerID="b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db" Oct 30 04:07:30.377588 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:30.377564 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db\": container with ID starting with b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db not found: ID does not exist" containerID="b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db" Oct 30 04:07:30.377659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.377600 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db"} err="failed to get container status \"b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db\": rpc error: code = NotFound desc = could not find container \"b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db\": container with ID starting with b8d018bc44095941d3a73b204a4c3a2d657813895cd2b79eade48dab9caba9db not found: ID does not exist" Oct 30 04:07:30.391989 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.391964 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs"] Oct 30 04:07:30.397374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.397351 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-da4553-openshift-default-6fcb679bdf-r54vs"] Oct 30 04:07:30.427104 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.427078 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk"] Oct 30 04:07:30.429417 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.429397 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-372480-67c4654f4d-z6cgk"] Oct 30 04:07:30.600366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:30.600294 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:31.573627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:31.573593 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm"] Oct 30 04:07:31.615383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:31.615354 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:31.717924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:31.717888 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:31.828388 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:31.828286 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:32.108295 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:32.108208 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7867e74-3d84-4780-88b8-a78222af9a95" path="/var/lib/kubelet/pods/d7867e74-3d84-4780-88b8-a78222af9a95/volumes" Oct 30 04:07:32.108689 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:32.108671 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" path="/var/lib/kubelet/pods/e59a5ab2-179c-47fc-a7c3-4dae630f458d/volumes" Oct 30 04:07:32.350184 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:32.350152 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:33.311989 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.311911 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:33.372646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.372612 4442 generic.go:356] "Generic (PLEG): container finished" podID="fd668d56-aefe-46e4-aa96-6d02036bac6f" containerID="d140c6dbf32eede2b07b23748914d78402a5cd4031abc844ac49c0048c95a0d0" exitCode=0 Oct 30 04:07:33.372824 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.372677 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" event={"ID":"fd668d56-aefe-46e4-aa96-6d02036bac6f","Type":"ContainerDied","Data":"d140c6dbf32eede2b07b23748914d78402a5cd4031abc844ac49c0048c95a0d0"} Oct 30 04:07:33.374692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.374668 4442 generic.go:356] "Generic (PLEG): container finished" podID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" containerID="a3c17e29e03104c1974eb7620f5751118fa3925131eec74a00f8652ad8c309f5" exitCode=0 Oct 30 04:07:33.374919 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.374901 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" event={"ID":"32bdd1c4-68d8-4426-8c82-fc5420f7b535","Type":"ContainerDied","Data":"a3c17e29e03104c1974eb7620f5751118fa3925131eec74a00f8652ad8c309f5"} Oct 30 04:07:33.430021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.429996 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 04:07:33.463202 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.463177 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 04:07:33.554005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.553970 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32bdd1c4-68d8-4426-8c82-fc5420f7b535-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bdd1c4-68d8-4426-8c82-fc5420f7b535-plugins-config-volume\") pod \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " Oct 30 04:07:33.554192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.554040 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32bdd1c4-68d8-4426-8c82-fc5420f7b535-kube-api-access-vt97t\" (UniqueName: \"kubernetes.io/projected/32bdd1c4-68d8-4426-8c82-fc5420f7b535-kube-api-access-vt97t\") pod \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\" (UID: \"32bdd1c4-68d8-4426-8c82-fc5420f7b535\") " Oct 30 04:07:33.554192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.554101 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd668d56-aefe-46e4-aa96-6d02036bac6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd668d56-aefe-46e4-aa96-6d02036bac6f-plugins-config-volume\") pod \"fd668d56-aefe-46e4-aa96-6d02036bac6f\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " Oct 30 04:07:33.554192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.554122 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd668d56-aefe-46e4-aa96-6d02036bac6f-kube-api-access-xgfpl\" (UniqueName: \"kubernetes.io/projected/fd668d56-aefe-46e4-aa96-6d02036bac6f-kube-api-access-xgfpl\") pod \"fd668d56-aefe-46e4-aa96-6d02036bac6f\" (UID: \"fd668d56-aefe-46e4-aa96-6d02036bac6f\") " Oct 30 04:07:33.554496 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.554469 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32bdd1c4-68d8-4426-8c82-fc5420f7b535-plugins-config-volume" pod "32bdd1c4-68d8-4426-8c82-fc5420f7b535" (UID: "32bdd1c4-68d8-4426-8c82-fc5420f7b535"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:33.554566 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.554490 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd668d56-aefe-46e4-aa96-6d02036bac6f-plugins-config-volume" pod "fd668d56-aefe-46e4-aa96-6d02036bac6f" (UID: "fd668d56-aefe-46e4-aa96-6d02036bac6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:33.554638 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.554623 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bdd1c4-68d8-4426-8c82-fc5420f7b535-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:33.554679 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.554641 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd668d56-aefe-46e4-aa96-6d02036bac6f-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:33.556308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.556287 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bdd1c4-68d8-4426-8c82-fc5420f7b535-kube-api-access-vt97t" pod "32bdd1c4-68d8-4426-8c82-fc5420f7b535" (UID: "32bdd1c4-68d8-4426-8c82-fc5420f7b535"). InnerVolumeSpecName "kube-api-access-vt97t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:33.556404 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.556326 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd668d56-aefe-46e4-aa96-6d02036bac6f-kube-api-access-xgfpl" pod "fd668d56-aefe-46e4-aa96-6d02036bac6f" (UID: "fd668d56-aefe-46e4-aa96-6d02036bac6f"). InnerVolumeSpecName "kube-api-access-xgfpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:33.655630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.655552 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt97t\" (UniqueName: \"kubernetes.io/projected/32bdd1c4-68d8-4426-8c82-fc5420f7b535-kube-api-access-vt97t\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:33.655630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.655579 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgfpl\" (UniqueName: \"kubernetes.io/projected/fd668d56-aefe-46e4-aa96-6d02036bac6f-kube-api-access-xgfpl\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:33.759966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:33.759895 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" podUID="7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" containerName="main" containerID="cri-o://6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69" gracePeriod=45 Oct 30 04:07:34.002264 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.002237 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 04:07:34.030395 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.030363 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:34.060152 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060125 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-model-cache\") pod \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " Oct 30 04:07:34.060348 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060183 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-home\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-home\") pod \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " Oct 30 04:07:34.060348 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060210 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-kube-api-access-9vk94\" (UniqueName: \"kubernetes.io/projected/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-kube-api-access-9vk94\") pod \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " Oct 30 04:07:34.060348 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060286 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-dshm\") pod \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " Oct 30 04:07:34.060547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060423 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-model-cache" pod "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" (UID: "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.060547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060487 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tls-certs\") pod \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " Oct 30 04:07:34.060547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060535 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tmp-dir\") pod \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\" (UID: \"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1\") " Oct 30 04:07:34.060990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.060961 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tmp-dir" pod "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" (UID: "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.063025 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.061391 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.063025 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.061419 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.063025 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.061643 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-home" pod "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" (UID: "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.063220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.063050 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tls-certs" pod "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" (UID: "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:34.063220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.063181 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-dshm" pod "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" (UID: "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:34.063350 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.063332 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-kube-api-access-9vk94" pod "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" (UID: "7e568b64-2cef-4f47-9ff6-fb8b4c576cf1"). InnerVolumeSpecName "kube-api-access-9vk94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:34.163007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.162972 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.163179 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.163009 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vk94\" (UniqueName: \"kubernetes.io/projected/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-kube-api-access-9vk94\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.163179 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.163026 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.163179 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.163039 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:34.382003 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.381971 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" Oct 30 04:07:34.382510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.381970 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9" event={"ID":"fd668d56-aefe-46e4-aa96-6d02036bac6f","Type":"ContainerDied","Data":"749a8937c550b220bfdf938edfffadd5b9b0f45b8f994d8e975c97bb7674428e"} Oct 30 04:07:34.382510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.382118 4442 scope.go:122] "RemoveContainer" containerID="d140c6dbf32eede2b07b23748914d78402a5cd4031abc844ac49c0048c95a0d0" Oct 30 04:07:34.383620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.383590 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" event={"ID":"32bdd1c4-68d8-4426-8c82-fc5420f7b535","Type":"ContainerDied","Data":"c6cb4c67552383ef49ef5effb3de454d3c75487d9c710cb6baaa15cc353fc38c"} Oct 30 04:07:34.383798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.383601 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr" Oct 30 04:07:34.385182 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.385156 4442 generic.go:356] "Generic (PLEG): container finished" podID="7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" containerID="6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69" exitCode=0 Oct 30 04:07:34.385281 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.385196 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" Oct 30 04:07:34.385336 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.385201 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" event={"ID":"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1","Type":"ContainerDied","Data":"6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69"} Oct 30 04:07:34.385374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.385356 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42" event={"ID":"7e568b64-2cef-4f47-9ff6-fb8b4c576cf1","Type":"ContainerDied","Data":"105e73945fbd2450afe79d52a3605931f95c104a7f5be3d97c66d2424fdf5e88"} Oct 30 04:07:34.392902 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.392847 4442 scope.go:122] "RemoveContainer" containerID="a3c17e29e03104c1974eb7620f5751118fa3925131eec74a00f8652ad8c309f5" Oct 30 04:07:34.401077 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.401062 4442 scope.go:122] "RemoveContainer" containerID="6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69" Oct 30 04:07:34.409533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.409516 4442 scope.go:122] "RemoveContainer" containerID="6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69" Oct 30 04:07:34.409819 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:34.409799 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69\": container with ID starting with 6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69 not found: ID does not exist" containerID="6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69" Oct 30 04:07:34.409891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.409829 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69"} err="failed to get container status \"6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69\": rpc error: code = NotFound desc = could not find container \"6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69\": container with ID starting with 6cde336f4cacb2a7513b4c5607fa11e9329a27edd85da8b504e7ff07d3ea2c69 not found: ID does not exist" Oct 30 04:07:34.429232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.429209 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42"] Oct 30 04:07:34.439742 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.439719 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a9c9b6/distinct-w0-a9c9b6-kserve-7bdf4d74c9-vwf42"] Oct 30 04:07:34.467739 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.467707 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9"] Oct 30 04:07:34.471589 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.471570 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-da4553-cb8bf6f67-7vnv9"] Oct 30 04:07:34.504873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.504850 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr"] Oct 30 04:07:34.508677 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:34.508654 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-da4553-55bd84f98-4vmmr"] Oct 30 04:07:35.467941 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:35.467912 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:36.107813 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.107774 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" path="/var/lib/kubelet/pods/32bdd1c4-68d8-4426-8c82-fc5420f7b535/volumes" Oct 30 04:07:36.108289 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.108269 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" path="/var/lib/kubelet/pods/7e568b64-2cef-4f47-9ff6-fb8b4c576cf1/volumes" Oct 30 04:07:36.108697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.108678 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd668d56-aefe-46e4-aa96-6d02036bac6f" path="/var/lib/kubelet/pods/fd668d56-aefe-46e4-aa96-6d02036bac6f/volumes" Oct 30 04:07:36.233215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.233186 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:36.333920 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.333884 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:36.574004 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.573953 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" podUID="5d629a0a-edeb-4351-9b6d-692d09939b9f" containerName="payload-processing" containerID="cri-o://35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603" gracePeriod=25 Oct 30 04:07:36.835153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.835040 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d"] Oct 30 04:07:36.835611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.835280 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" podUID="4ef1519e-576f-41a0-9a11-9596736afc1b" containerName="maas-api" containerID="cri-o://27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b" gracePeriod=30 Oct 30 04:07:36.895978 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:36.895943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:37.203802 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.195506 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 04:07:37.319024 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.318987 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls\") pod \"4ef1519e-576f-41a0-9a11-9596736afc1b\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " Oct 30 04:07:37.319465 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.319435 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-metrics-certs\") pod \"4ef1519e-576f-41a0-9a11-9596736afc1b\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " Oct 30 04:07:37.319803 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.319781 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef1519e-576f-41a0-9a11-9596736afc1b-kube-api-access-v986p\" (UniqueName: \"kubernetes.io/projected/4ef1519e-576f-41a0-9a11-9596736afc1b-kube-api-access-v986p\") pod \"4ef1519e-576f-41a0-9a11-9596736afc1b\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " Oct 30 04:07:37.319979 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.319959 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef1519e-576f-41a0-9a11-9596736afc1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef1519e-576f-41a0-9a11-9596736afc1b-openshift-service-ca\") pod \"4ef1519e-576f-41a0-9a11-9596736afc1b\" (UID: \"4ef1519e-576f-41a0-9a11-9596736afc1b\") " Oct 30 04:07:37.320733 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.320383 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef1519e-576f-41a0-9a11-9596736afc1b-openshift-service-ca" pod "4ef1519e-576f-41a0-9a11-9596736afc1b" (UID: "4ef1519e-576f-41a0-9a11-9596736afc1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:37.320733 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.320694 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ef1519e-576f-41a0-9a11-9596736afc1b-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:37.328133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.328106 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-metrics-certs" pod "4ef1519e-576f-41a0-9a11-9596736afc1b" (UID: "4ef1519e-576f-41a0-9a11-9596736afc1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:37.328489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.328464 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef1519e-576f-41a0-9a11-9596736afc1b-kube-api-access-v986p" pod "4ef1519e-576f-41a0-9a11-9596736afc1b" (UID: "4ef1519e-576f-41a0-9a11-9596736afc1b"). InnerVolumeSpecName "kube-api-access-v986p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:37.331894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.331859 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls" pod "4ef1519e-576f-41a0-9a11-9596736afc1b" (UID: "4ef1519e-576f-41a0-9a11-9596736afc1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:37.376400 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.376368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:37.406087 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.406053 4442 generic.go:356] "Generic (PLEG): container finished" podID="4ef1519e-576f-41a0-9a11-9596736afc1b" containerID="27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b" exitCode=0 Oct 30 04:07:37.406258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.406119 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" Oct 30 04:07:37.406258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.406136 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" event={"ID":"4ef1519e-576f-41a0-9a11-9596736afc1b","Type":"ContainerDied","Data":"27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b"} Oct 30 04:07:37.406258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.406194 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d" event={"ID":"4ef1519e-576f-41a0-9a11-9596736afc1b","Type":"ContainerDied","Data":"7730fc5cebbf34405dc72e4eca7a53bb04a71b1f875b57968e4efe958d29d852"} Oct 30 04:07:37.406258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.406218 4442 scope.go:122] "RemoveContainer" containerID="27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b" Oct 30 04:07:37.418435 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.418416 4442 scope.go:122] "RemoveContainer" containerID="27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b" Oct 30 04:07:37.418712 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:37.418692 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b\": container with ID starting with 27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b not found: ID does not exist" containerID="27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b" Oct 30 04:07:37.418841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.418726 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b"} err="failed to get container status \"27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b\": rpc error: code = NotFound desc = could not find container \"27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b\": container with ID starting with 27b42dfff2d83e02a20232380c8a72ebe4e6f0ed1f1d722c12c981de44cc216b not found: ID does not exist" Oct 30 04:07:37.422068 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.422046 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:37.422145 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.422070 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v986p\" (UniqueName: \"kubernetes.io/projected/4ef1519e-576f-41a0-9a11-9596736afc1b-kube-api-access-v986p\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:37.422145 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.422080 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ef1519e-576f-41a0-9a11-9596736afc1b-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:37.485999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.485811 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:37.491449 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.491411 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d"] Oct 30 04:07:37.495105 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.495083 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a9c9b6-59f5574b86-pwf5d"] Oct 30 04:07:37.593659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.593619 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:37.758655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:37.758626 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:38.108495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:38.108425 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef1519e-576f-41a0-9a11-9596736afc1b" path="/var/lib/kubelet/pods/4ef1519e-576f-41a0-9a11-9596736afc1b/volumes" Oct 30 04:07:38.942568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:38.942525 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:39.224174 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:39.224073 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" podUID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:39.260337 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:39.260304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:39.922455 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:39.922424 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:40.043469 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.043395 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 04:07:40.075948 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.075918 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 04:07:40.149378 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.149338 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:40.150392 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.150366 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c7cd60-845b-4415-a560-1e913027eeb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c7cd60-845b-4415-a560-1e913027eeb6-plugins-config-volume\") pod \"60c7cd60-845b-4415-a560-1e913027eeb6\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " Oct 30 04:07:40.150515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.150430 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-kube-api-access-c7tr7\" (UniqueName: \"kubernetes.io/projected/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-kube-api-access-c7tr7\") pod \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " Oct 30 04:07:40.150655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.150534 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-plugins-config-volume\") pod \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\" (UID: \"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca\") " Oct 30 04:07:40.150718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.150659 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c7cd60-845b-4415-a560-1e913027eeb6-kube-api-access-pwkqf\" (UniqueName: \"kubernetes.io/projected/60c7cd60-845b-4415-a560-1e913027eeb6-kube-api-access-pwkqf\") pod \"60c7cd60-845b-4415-a560-1e913027eeb6\" (UID: \"60c7cd60-845b-4415-a560-1e913027eeb6\") " Oct 30 04:07:40.150936 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.150915 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c7cd60-845b-4415-a560-1e913027eeb6-plugins-config-volume" pod "60c7cd60-845b-4415-a560-1e913027eeb6" (UID: "60c7cd60-845b-4415-a560-1e913027eeb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:40.151026 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.150973 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-plugins-config-volume" pod "f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" (UID: "f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:40.151118 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.151095 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:40.151118 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.151115 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c7cd60-845b-4415-a560-1e913027eeb6-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:40.152898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.152873 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c7cd60-845b-4415-a560-1e913027eeb6-kube-api-access-pwkqf" pod "60c7cd60-845b-4415-a560-1e913027eeb6" (UID: "60c7cd60-845b-4415-a560-1e913027eeb6"). InnerVolumeSpecName "kube-api-access-pwkqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:40.153344 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.153326 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-kube-api-access-c7tr7" pod "f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" (UID: "f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca"). InnerVolumeSpecName "kube-api-access-c7tr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:40.236045 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.235992 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:40.252207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.252175 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwkqf\" (UniqueName: \"kubernetes.io/projected/60c7cd60-845b-4415-a560-1e913027eeb6-kube-api-access-pwkqf\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:40.252207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.252205 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7tr7\" (UniqueName: \"kubernetes.io/projected/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca-kube-api-access-c7tr7\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:40.363632 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.363564 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:40.457226 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.457185 4442 generic.go:356] "Generic (PLEG): container finished" podID="60c7cd60-845b-4415-a560-1e913027eeb6" containerID="c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723" exitCode=0 Oct 30 04:07:40.457443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.457318 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" event={"ID":"60c7cd60-845b-4415-a560-1e913027eeb6","Type":"ContainerDied","Data":"c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723"} Oct 30 04:07:40.457443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.457338 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" Oct 30 04:07:40.457443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.457373 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2" event={"ID":"60c7cd60-845b-4415-a560-1e913027eeb6","Type":"ContainerDied","Data":"34cf5ab63c147652aa56f29d101e9492f6f67c8d5ec1119e34396199d95ce291"} Oct 30 04:07:40.457443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.457403 4442 scope.go:122] "RemoveContainer" containerID="c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723" Oct 30 04:07:40.459459 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.459432 4442 generic.go:356] "Generic (PLEG): container finished" podID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" containerID="598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4" exitCode=0 Oct 30 04:07:40.459609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.459478 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" Oct 30 04:07:40.459609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.459484 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" event={"ID":"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca","Type":"ContainerDied","Data":"598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4"} Oct 30 04:07:40.459609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.459526 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj" event={"ID":"f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca","Type":"ContainerDied","Data":"c5ec2ec057752f5e8cabba913200f8eba04d970423c9f0353698380fef6600ef"} Oct 30 04:07:40.463808 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.463738 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:40.479018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.476099 4442 scope.go:122] "RemoveContainer" containerID="c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723" Oct 30 04:07:40.479018 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:40.476825 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723\": container with ID starting with c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723 not found: ID does not exist" containerID="c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723" Oct 30 04:07:40.479018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.476859 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723"} err="failed to get container status \"c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723\": rpc error: code = NotFound desc = could not find container \"c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723\": container with ID starting with c4827753406b3be5d3e5c452908f98edf916f210ae0a67a9df82ab2880dba723 not found: ID does not exist" Oct 30 04:07:40.479018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.476884 4442 scope.go:122] "RemoveContainer" containerID="598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4" Oct 30 04:07:40.486203 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.486181 4442 scope.go:122] "RemoveContainer" containerID="598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4" Oct 30 04:07:40.486471 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:40.486451 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4\": container with ID starting with 598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4 not found: ID does not exist" containerID="598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4" Oct 30 04:07:40.486520 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.486478 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4"} err="failed to get container status \"598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4\": rpc error: code = NotFound desc = could not find container \"598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4\": container with ID starting with 598b01b0c753966aa1bad516a3f8ba14d03e51e0ab650432d76e52f2c661b0b4 not found: ID does not exist" Oct 30 04:07:40.511289 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.511265 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2"] Oct 30 04:07:40.515703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.515677 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-372480-7589999f7d-td2g2"] Oct 30 04:07:40.544961 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.544937 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj"] Oct 30 04:07:40.548562 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.548543 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-372480-7cd44b98c-cnzzj"] Oct 30 04:07:40.688580 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.688498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:40.849867 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:40.849835 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:41.279509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:41.279478 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:41.380778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:41.379802 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:41.728567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:41.728484 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5"] Oct 30 04:07:41.728797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:41.728735 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" containerID="cri-o://7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d" gracePeriod=30 Oct 30 04:07:41.731173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:41.730012 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:42.077259 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:42.077227 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:42.106652 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:42.106621 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c7cd60-845b-4415-a560-1e913027eeb6" path="/var/lib/kubelet/pods/60c7cd60-845b-4415-a560-1e913027eeb6/volumes" Oct 30 04:07:42.107055 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:42.107041 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" path="/var/lib/kubelet/pods/f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca/volumes" Oct 30 04:07:42.448834 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:42.448308 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:43.783924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:43.783890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:43.869020 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:43.868990 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:44.350542 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:44.350508 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:07:45.331723 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:45.331685 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" podUID="5d629a0a-edeb-4351-9b6d-692d09939b9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:45.560610 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:45.560574 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:46.998809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:46.998785 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 04:07:47.063581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063500 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-token\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-token\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.063581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063545 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-credential-socket\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.063869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063602 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-certs\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.063869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063642 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-envoy\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.063869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063700 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-kube-api-access-hfbf4\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-kube-api-access-hfbf4\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.063869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063738 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-podinfo\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.063869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063790 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-data\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.063869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063858 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bd0484-a942-4599-baa2-c8d1bb27e767-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bd0484-a942-4599-baa2-c8d1bb27e767-istiod-ca-cert\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.064160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063910 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-socket\") pod \"a8bd0484-a942-4599-baa2-c8d1bb27e767\" (UID: \"a8bd0484-a942-4599-baa2-c8d1bb27e767\") " Oct 30 04:07:47.064160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.064026 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-certs" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:47.064160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.063984 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-credential-socket" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:47.064548 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.064325 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bd0484-a942-4599-baa2-c8d1bb27e767-istiod-ca-cert" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:47.064548 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.064427 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.064548 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.064445 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.064548 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.064511 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-data" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:47.064843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.064564 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-socket" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:47.066485 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.066214 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-podinfo" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:07:47.066485 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.066289 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-token" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:47.066974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.066948 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-kube-api-access-hfbf4" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "kube-api-access-hfbf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:47.069076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.069044 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-envoy" pod "a8bd0484-a942-4599-baa2-c8d1bb27e767" (UID: "a8bd0484-a942-4599-baa2-c8d1bb27e767"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:07:47.166188 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.166151 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8bd0484-a942-4599-baa2-c8d1bb27e767-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.166188 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.166185 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.166461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.166200 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.166461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.166214 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.166461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.166228 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfbf4\" (UniqueName: \"kubernetes.io/projected/a8bd0484-a942-4599-baa2-c8d1bb27e767-kube-api-access-hfbf4\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.166461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.166242 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.166461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.166255 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8bd0484-a942-4599-baa2-c8d1bb27e767-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:47.318577 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.318491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:47.507843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.507809 4442 generic.go:356] "Generic (PLEG): container finished" podID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerID="7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d" exitCode=0 Oct 30 04:07:47.508039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.507856 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" Oct 30 04:07:47.508039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.507894 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" event={"ID":"a8bd0484-a942-4599-baa2-c8d1bb27e767","Type":"ContainerDied","Data":"7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d"} Oct 30 04:07:47.508039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.507954 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5" event={"ID":"a8bd0484-a942-4599-baa2-c8d1bb27e767","Type":"ContainerDied","Data":"0f36608925416a28b03e38472874b2ec35cc6dfdeef86c4584388d28508bd7b1"} Oct 30 04:07:47.508039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.507980 4442 scope.go:122] "RemoveContainer" containerID="7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d" Oct 30 04:07:47.519489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.519471 4442 scope.go:122] "RemoveContainer" containerID="7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d" Oct 30 04:07:47.519798 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:47.519778 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d\": container with ID starting with 7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d not found: ID does not exist" containerID="7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d" Oct 30 04:07:47.519875 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.519805 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d"} err="failed to get container status \"7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d\": rpc error: code = NotFound desc = could not find container \"7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d\": container with ID starting with 7a6105e716b705df634f8d68482c20d0c89785980ab7b4b83c1db8ecc155ae2d not found: ID does not exist" Oct 30 04:07:47.555017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.554991 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5"] Oct 30 04:07:47.566290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:47.566260 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a9c9b6-openshift-default-6fd8dbdb4f-rvkl5"] Oct 30 04:07:48.107372 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:48.107340 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" path="/var/lib/kubelet/pods/a8bd0484-a942-4599-baa2-c8d1bb27e767/volumes" Oct 30 04:07:48.612728 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:48.612699 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96"] Oct 30 04:07:48.612982 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:48.612957 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" podUID="ac4a8fc7-1451-454a-979d-e1f05a3e1570" containerName="maas-api" containerID="cri-o://a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a" gracePeriod=30 Oct 30 04:07:48.667066 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:48.667031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:48.916454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:48.916429 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 04:07:49.026797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.026481 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-maas-api-tls\") pod \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " Oct 30 04:07:49.026797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.026551 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4a8fc7-1451-454a-979d-e1f05a3e1570-kube-api-access-phtw8\" (UniqueName: \"kubernetes.io/projected/ac4a8fc7-1451-454a-979d-e1f05a3e1570-kube-api-access-phtw8\") pod \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " Oct 30 04:07:49.026797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.026580 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-metrics-certs\") pod \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " Oct 30 04:07:49.026797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.026623 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4a8fc7-1451-454a-979d-e1f05a3e1570-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4a8fc7-1451-454a-979d-e1f05a3e1570-openshift-service-ca\") pod \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\" (UID: \"ac4a8fc7-1451-454a-979d-e1f05a3e1570\") " Oct 30 04:07:49.045370 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.045089 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-maas-api-tls" pod "ac4a8fc7-1451-454a-979d-e1f05a3e1570" (UID: "ac4a8fc7-1451-454a-979d-e1f05a3e1570"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:49.064196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.064093 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4a8fc7-1451-454a-979d-e1f05a3e1570-kube-api-access-phtw8" pod "ac4a8fc7-1451-454a-979d-e1f05a3e1570" (UID: "ac4a8fc7-1451-454a-979d-e1f05a3e1570"). InnerVolumeSpecName "kube-api-access-phtw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:49.074199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.074164 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4a8fc7-1451-454a-979d-e1f05a3e1570-openshift-service-ca" pod "ac4a8fc7-1451-454a-979d-e1f05a3e1570" (UID: "ac4a8fc7-1451-454a-979d-e1f05a3e1570"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:49.076233 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.076175 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-metrics-certs" pod "ac4a8fc7-1451-454a-979d-e1f05a3e1570" (UID: "ac4a8fc7-1451-454a-979d-e1f05a3e1570"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:07:49.077601 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.077575 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:49.128521 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.128491 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:49.128521 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.128520 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtw8\" (UniqueName: \"kubernetes.io/projected/ac4a8fc7-1451-454a-979d-e1f05a3e1570-kube-api-access-phtw8\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:49.128521 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.128530 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac4a8fc7-1451-454a-979d-e1f05a3e1570-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:49.128983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.128538 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac4a8fc7-1451-454a-979d-e1f05a3e1570-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:49.172696 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.172624 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:49.345135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.345099 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:49.523351 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.523317 4442 generic.go:356] "Generic (PLEG): container finished" podID="ac4a8fc7-1451-454a-979d-e1f05a3e1570" containerID="a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a" exitCode=0 Oct 30 04:07:49.523539 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.523367 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" Oct 30 04:07:49.523539 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.523384 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" event={"ID":"ac4a8fc7-1451-454a-979d-e1f05a3e1570","Type":"ContainerDied","Data":"a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a"} Oct 30 04:07:49.523539 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.523440 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96" event={"ID":"ac4a8fc7-1451-454a-979d-e1f05a3e1570","Type":"ContainerDied","Data":"5a2979a2f746f0ab8f7ac6a6b3ecd1b0377630c70fc65abe609fff33292b1c44"} Oct 30 04:07:49.523539 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.523458 4442 scope.go:122] "RemoveContainer" containerID="a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a" Oct 30 04:07:49.533781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.533764 4442 scope.go:122] "RemoveContainer" containerID="a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a" Oct 30 04:07:49.534089 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:49.534071 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a\": container with ID starting with a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a not found: ID does not exist" containerID="a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a" Oct 30 04:07:49.534144 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.534097 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a"} err="failed to get container status \"a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a\": rpc error: code = NotFound desc = could not find container \"a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a\": container with ID starting with a2a4234dae681ffa5f7463365430ffc89cd3aea2851798c1e2e7bca3c606491a not found: ID does not exist" Oct 30 04:07:49.571722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.571700 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96"] Oct 30 04:07:49.582454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.582422 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e1a0c2-76f5b7dd7c-zjx96"] Oct 30 04:07:49.605625 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:49.605596 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:50.106712 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:50.106680 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4a8fc7-1451-454a-979d-e1f05a3e1570" path="/var/lib/kubelet/pods/ac4a8fc7-1451-454a-979d-e1f05a3e1570/volumes" Oct 30 04:07:50.367483 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:50.367387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:50.493954 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:50.489299 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:50.669124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:50.669043 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:51.295942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:51.295900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:52.570702 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:52.570669 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:52.693913 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:52.693880 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:53.579701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:53.579670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:54.381046 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:54.381016 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:54.861074 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:54.861035 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:07:54.929660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:54.929630 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:55.031337 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:55.031301 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:55.332080 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:55.332041 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" podUID="5d629a0a-edeb-4351-9b6d-692d09939b9f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:07:55.342118 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:55.342090 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:55.563474 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:55.563430 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:55.926235 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:55.926196 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq"] Oct 30 04:07:55.926684 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:55.926569 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" containerID="cri-o://330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8" gracePeriod=30 Oct 30 04:07:55.953476 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:55.953448 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:56.178263 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.178185 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:56.180439 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.180399 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:07:56.180603 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.180471 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:07:56.278480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.278442 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:56.433257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.433158 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:56.795400 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.795372 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:07:56.801340 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.801322 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:07:56.838622 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.838601 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 04:07:56.900065 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.899987 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:56.937630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.937599 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d629a0a-edeb-4351-9b6d-692d09939b9f-kube-api-access-mn4jz\" (UniqueName: \"kubernetes.io/projected/5d629a0a-edeb-4351-9b6d-692d09939b9f-kube-api-access-mn4jz\") pod \"5d629a0a-edeb-4351-9b6d-692d09939b9f\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " Oct 30 04:07:56.938127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.938102 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d629a0a-edeb-4351-9b6d-692d09939b9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d629a0a-edeb-4351-9b6d-692d09939b9f-plugins-config-volume\") pod \"5d629a0a-edeb-4351-9b6d-692d09939b9f\" (UID: \"5d629a0a-edeb-4351-9b6d-692d09939b9f\") " Oct 30 04:07:56.938597 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.938566 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d629a0a-edeb-4351-9b6d-692d09939b9f-plugins-config-volume" pod "5d629a0a-edeb-4351-9b6d-692d09939b9f" (UID: "5d629a0a-edeb-4351-9b6d-692d09939b9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:07:56.940326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:56.940305 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d629a0a-edeb-4351-9b6d-692d09939b9f-kube-api-access-mn4jz" pod "5d629a0a-edeb-4351-9b6d-692d09939b9f" (UID: "5d629a0a-edeb-4351-9b6d-692d09939b9f"). InnerVolumeSpecName "kube-api-access-mn4jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:07:57.040239 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.040212 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d629a0a-edeb-4351-9b6d-692d09939b9f-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:57.040239 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.040237 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn4jz\" (UniqueName: \"kubernetes.io/projected/5d629a0a-edeb-4351-9b6d-692d09939b9f-kube-api-access-mn4jz\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:07:57.085421 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.085383 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:57.589430 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.589392 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:57.597107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.597073 4442 generic.go:356] "Generic (PLEG): container finished" podID="5d629a0a-edeb-4351-9b6d-692d09939b9f" containerID="35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603" exitCode=0 Oct 30 04:07:57.597292 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.597140 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" Oct 30 04:07:57.597292 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.597171 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" event={"ID":"5d629a0a-edeb-4351-9b6d-692d09939b9f","Type":"ContainerDied","Data":"35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603"} Oct 30 04:07:57.597292 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.597230 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm" event={"ID":"5d629a0a-edeb-4351-9b6d-692d09939b9f","Type":"ContainerDied","Data":"1b114e0c491faa7bb8445b1811ba4db4ac375162393a2caf9f4942c498ed19dc"} Oct 30 04:07:57.597292 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.597265 4442 scope.go:122] "RemoveContainer" containerID="35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603" Oct 30 04:07:57.608944 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.608925 4442 scope.go:122] "RemoveContainer" containerID="35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603" Oct 30 04:07:57.609229 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:07:57.609213 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603\": container with ID starting with 35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603 not found: ID does not exist" containerID="35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603" Oct 30 04:07:57.609276 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.609236 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603"} err="failed to get container status \"35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603\": rpc error: code = NotFound desc = could not find container \"35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603\": container with ID starting with 35c93f7549144c413444be7dd2f3cb12f51c361f3c1813b8598cdddc6d9dc603 not found: ID does not exist" Oct 30 04:07:57.660051 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.660017 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm"] Oct 30 04:07:57.669794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.669743 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e1a0c2-7cdd789b76-fl8sm"] Oct 30 04:07:57.691056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:57.691021 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:58.015142 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:58.015107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:58.029971 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:58.029936 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9"] Oct 30 04:07:58.030286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:58.030197 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" containerID="cri-o://3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce" gracePeriod=30 Oct 30 04:07:58.109587 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:58.109483 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d629a0a-edeb-4351-9b6d-692d09939b9f" path="/var/lib/kubelet/pods/5d629a0a-edeb-4351-9b6d-692d09939b9f/volumes" Oct 30 04:07:58.112998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:58.112974 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:58.923982 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:58.923943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:59.708565 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:07:59.708529 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:01.114697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.114661 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:01.209658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.209586 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 04:08:01.295900 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.295865 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-socket\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.295964 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-envoy\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.295995 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-podinfo\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.296043 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-token\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-token\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.296065 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb022099-cfd2-41b8-aabc-c6cb77f40112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb022099-cfd2-41b8-aabc-c6cb77f40112-istiod-ca-cert\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.296085 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-kube-api-access-cqpqz\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-kube-api-access-cqpqz\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.296117 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-data\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.296135 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-certs\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.296345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.296165 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-credential-socket\") pod \"cb022099-cfd2-41b8-aabc-c6cb77f40112\" (UID: \"cb022099-cfd2-41b8-aabc-c6cb77f40112\") " Oct 30 04:08:01.299418 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.299219 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-socket" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.300206 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.299994 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-data" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.301522 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.301473 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb022099-cfd2-41b8-aabc-c6cb77f40112-istiod-ca-cert" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:01.302668 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.302229 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-envoy" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.302668 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.302299 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-token" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:01.302910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.302823 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-podinfo" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:08:01.303172 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.303063 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-certs" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.303766 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.303716 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-credential-socket" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:01.304545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.304471 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.304545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.304500 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.304854 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.304525 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.304854 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.304803 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb022099-cfd2-41b8-aabc-c6cb77f40112-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.304854 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.304818 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.305251 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.304832 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.305251 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.305208 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.305571 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.305232 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb022099-cfd2-41b8-aabc-c6cb77f40112-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.319910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.319867 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-kube-api-access-cqpqz" pod "cb022099-cfd2-41b8-aabc-c6cb77f40112" (UID: "cb022099-cfd2-41b8-aabc-c6cb77f40112"). InnerVolumeSpecName "kube-api-access-cqpqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:01.406580 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.406542 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqpqz\" (UniqueName: \"kubernetes.io/projected/cb022099-cfd2-41b8-aabc-c6cb77f40112-kube-api-access-cqpqz\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:01.517589 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.517554 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:01.620040 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.620010 4442 generic.go:356] "Generic (PLEG): container finished" podID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerID="330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8" exitCode=0 Oct 30 04:08:01.620209 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.620064 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" event={"ID":"cb022099-cfd2-41b8-aabc-c6cb77f40112","Type":"ContainerDied","Data":"330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8"} Oct 30 04:08:01.620209 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.620081 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" Oct 30 04:08:01.620209 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.620129 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq" event={"ID":"cb022099-cfd2-41b8-aabc-c6cb77f40112","Type":"ContainerDied","Data":"39c6894f56a0124493642b972a6f6db59375839d53e794cb4735ac68ebefdc45"} Oct 30 04:08:01.620209 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.620151 4442 scope.go:122] "RemoveContainer" containerID="330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8" Oct 30 04:08:01.630549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.630528 4442 scope.go:122] "RemoveContainer" containerID="f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff" Oct 30 04:08:01.639537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.639517 4442 scope.go:122] "RemoveContainer" containerID="330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8" Oct 30 04:08:01.639857 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:01.639836 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8\": container with ID starting with 330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8 not found: ID does not exist" containerID="330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8" Oct 30 04:08:01.639940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.639867 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8"} err="failed to get container status \"330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8\": rpc error: code = NotFound desc = could not find container \"330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8\": container with ID starting with 330a9e5ae86f80557eafd0a64e2c7888ba42233bf6bb72fa470130e2556343b8 not found: ID does not exist" Oct 30 04:08:01.639940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.639885 4442 scope.go:122] "RemoveContainer" containerID="f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff" Oct 30 04:08:01.640135 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:01.640116 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff\": container with ID starting with f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff not found: ID does not exist" containerID="f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff" Oct 30 04:08:01.640178 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.640142 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff"} err="failed to get container status \"f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff\": rpc error: code = NotFound desc = could not find container \"f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff\": container with ID starting with f7b2381302d0aeaf2ae789d688c96b060f80d03b2ce785e517885aa1b24a57ff not found: ID does not exist" Oct 30 04:08:01.667679 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.667641 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq"] Oct 30 04:08:01.678370 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:01.678341 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e1a0c2-openshift-default-79f796bf87-jfnsq"] Oct 30 04:08:02.106216 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:02.106185 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" path="/var/lib/kubelet/pods/cb022099-cfd2-41b8-aabc-c6cb77f40112/volumes" Oct 30 04:08:02.692503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:02.692465 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:03.308260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.308201 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:08:03.328492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328460 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-podinfo\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328510 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-credential-socket\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328532 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-kube-api-access-6m6x4\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-kube-api-access-6m6x4\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328552 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-socket\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328572 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-envoy\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328607 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istiod-ca-cert\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328937 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328671 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-token\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-token\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328937 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328717 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-certs\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.328937 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328842 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-data\") pod \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\" (UID: \"3f831b01-41b2-4f28-b2b4-b33568d8c70e\") " Oct 30 04:08:03.329107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328905 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-credential-socket" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:03.329107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.328980 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-socket" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:03.329225 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.329118 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-certs" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:03.329293 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.329272 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-data" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:03.329388 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.329355 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.329388 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.329380 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.329388 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.329393 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.329794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.329543 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istiod-ca-cert" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:03.331037 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.331012 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-podinfo" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:08:03.331168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.331033 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-envoy" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:03.331236 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.331180 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-kube-api-access-6m6x4" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "kube-api-access-6m6x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:03.331426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.331408 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-token" pod "3f831b01-41b2-4f28-b2b4-b33568d8c70e" (UID: "3f831b01-41b2-4f28-b2b4-b33568d8c70e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:03.430932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.430900 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.430932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.430927 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.430932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.430935 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.431257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.430943 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.431257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.430951 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m6x4\" (UniqueName: \"kubernetes.io/projected/3f831b01-41b2-4f28-b2b4-b33568d8c70e-kube-api-access-6m6x4\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.431257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.430960 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f831b01-41b2-4f28-b2b4-b33568d8c70e-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:03.633429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.633335 4442 generic.go:356] "Generic (PLEG): container finished" podID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerID="3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce" exitCode=0 Oct 30 04:08:03.633429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.633382 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" Oct 30 04:08:03.633429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.633419 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" event={"ID":"3f831b01-41b2-4f28-b2b4-b33568d8c70e","Type":"ContainerDied","Data":"3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce"} Oct 30 04:08:03.633890 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.633471 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9" event={"ID":"3f831b01-41b2-4f28-b2b4-b33568d8c70e","Type":"ContainerDied","Data":"eb00fb4a60f1c482a05203fdd1da7f2ab8585ce6a9694c0758b785bf043db3d0"} Oct 30 04:08:03.633890 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.633494 4442 scope.go:122] "RemoveContainer" containerID="3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce" Oct 30 04:08:03.643937 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.643916 4442 scope.go:122] "RemoveContainer" containerID="3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce" Oct 30 04:08:03.644197 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:03.644180 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce\": container with ID starting with 3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce not found: ID does not exist" containerID="3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce" Oct 30 04:08:03.644253 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.644206 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce"} err="failed to get container status \"3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce\": rpc error: code = NotFound desc = could not find container \"3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce\": container with ID starting with 3b0866c2dc3ad70d6f2a9c2f75cd2722dc36bdccc9df1f0a15996c93bd721dce not found: ID does not exist" Oct 30 04:08:03.680380 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.680350 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9"] Oct 30 04:08:03.690917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.690884 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-7d99a4-openshift-default-5f5b5d6bd7-sqsp9"] Oct 30 04:08:03.972454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:03.972367 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:04.106698 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:04.106662 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" path="/var/lib/kubelet/pods/3f831b01-41b2-4f28-b2b4-b33568d8c70e/volumes" Oct 30 04:08:04.193229 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:04.193197 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:04.274468 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:04.274428 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:08:04.281481 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:04.281454 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p"] Oct 30 04:08:04.336279 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:04.336247 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6"] Oct 30 04:08:04.448918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:04.448888 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:04.549551 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:04.549412 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:05.890153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:05.890119 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:06.270504 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:06.270452 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:06.510344 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:06.510311 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:07.114305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114275 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z"] Oct 30 04:08:07.114771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114739 4442 cpu_manager.go:414] "containerMap: removing container" podUID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" containerName="payload-pre-processing" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114771 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" containerName="payload-pre-processing" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114785 4442 cpu_manager.go:414] "containerMap: removing container" podUID="15707df2-3b1c-43a8-8a99-b1f10231cb00" containerName="maas-api" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114790 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15707df2-3b1c-43a8-8a99-b1f10231cb00" containerName="maas-api" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114797 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2c775e5d-3134-40d3-8d81-4748781896a5" containerName="payload-processing" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114802 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c775e5d-3134-40d3-8d81-4748781896a5" containerName="payload-processing" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114813 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114818 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114825 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114830 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114836 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" containerName="payload-pre-processing" Oct 30 04:08:07.114836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114840 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" containerName="payload-pre-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114846 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114850 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114859 4442 cpu_manager.go:414] "containerMap: removing container" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114864 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114872 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114876 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114883 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7e4351d9-3c06-4da5-96a8-11ffd833984a" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114887 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4351d9-3c06-4da5-96a8-11ffd833984a" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114893 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114897 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114906 4442 cpu_manager.go:414] "containerMap: removing container" podUID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114910 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114916 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d7867e74-3d84-4780-88b8-a78222af9a95" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114920 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7867e74-3d84-4780-88b8-a78222af9a95" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114928 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fd668d56-aefe-46e4-aa96-6d02036bac6f" containerName="payload-pre-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114932 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd668d56-aefe-46e4-aa96-6d02036bac6f" containerName="payload-pre-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114941 4442 cpu_manager.go:414] "containerMap: removing container" podUID="452ef5f7-56c5-4b2b-863c-58414354fca5" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114945 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452ef5f7-56c5-4b2b-863c-58414354fca5" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114955 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bd6517be-64ee-4ab8-97e0-26b4d01784f6" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114959 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6517be-64ee-4ab8-97e0-26b4d01784f6" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114966 4442 cpu_manager.go:414] "containerMap: removing container" podUID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114970 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114978 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2b37ede5-6f78-44aa-9211-4e23d05941c2" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114983 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b37ede5-6f78-44aa-9211-4e23d05941c2" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114990 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5d629a0a-edeb-4351-9b6d-692d09939b9f" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.114994 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d629a0a-edeb-4351-9b6d-692d09939b9f" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115000 4442 cpu_manager.go:414] "containerMap: removing container" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115005 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115014 4442 cpu_manager.go:414] "containerMap: removing container" podUID="38351d05-2730-4456-8bb6-46226286b4c1" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115019 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38351d05-2730-4456-8bb6-46226286b4c1" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115026 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ac4a8fc7-1451-454a-979d-e1f05a3e1570" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115031 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4a8fc7-1451-454a-979d-e1f05a3e1570" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115037 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7b752eca-931a-4222-8569-8bae91568839" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115042 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b752eca-931a-4222-8569-8bae91568839" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115047 4442 cpu_manager.go:414] "containerMap: removing container" podUID="60c7cd60-845b-4415-a560-1e913027eeb6" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115051 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c7cd60-845b-4415-a560-1e913027eeb6" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115057 4442 cpu_manager.go:414] "containerMap: removing container" podUID="4ef1519e-576f-41a0-9a11-9596736afc1b" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115061 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef1519e-576f-41a0-9a11-9596736afc1b" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115128 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bd0484-a942-4599-baa2-c8d1bb27e767" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115135 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f993ec35-1ad1-4ab0-b5a4-eebc93dd66ca" containerName="payload-pre-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115142 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30726b9-6954-4842-8ce1-c7c20fc326f1" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115149 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59a5ab2-179c-47fc-a7c3-4dae630f458d" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115155 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e568b64-2cef-4f47-9ff6-fb8b4c576cf1" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115161 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4351d9-3c06-4da5-96a8-11ffd833984a" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115166 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f831b01-41b2-4f28-b2b4-b33568d8c70e" containerName="istio-proxy" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115171 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="acdb7dd7-0fa4-45e8-995e-2ec2238049c5" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115176 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7867e74-3d84-4780-88b8-a78222af9a95" containerName="maas-api" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115181 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd668d56-aefe-46e4-aa96-6d02036bac6f" containerName="payload-pre-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115188 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="452ef5f7-56c5-4b2b-863c-58414354fca5" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115194 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6517be-64ee-4ab8-97e0-26b4d01784f6" containerName="main" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115200 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bdd1c4-68d8-4426-8c82-fc5420f7b535" containerName="payload-processing" Oct 30 04:08:07.115186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115206 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b37ede5-6f78-44aa-9211-4e23d05941c2" containerName="payload-processing" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115212 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d629a0a-edeb-4351-9b6d-692d09939b9f" containerName="payload-processing" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115216 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="723efe3e-62f7-48ba-b1ed-c15032afd156" containerName="istio-proxy" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115222 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="38351d05-2730-4456-8bb6-46226286b4c1" containerName="maas-api" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115227 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4a8fc7-1451-454a-979d-e1f05a3e1570" containerName="maas-api" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115232 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115236 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b752eca-931a-4222-8569-8bae91568839" containerName="main" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115241 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c7cd60-845b-4415-a560-1e913027eeb6" containerName="payload-processing" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115246 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef1519e-576f-41a0-9a11-9596736afc1b" containerName="maas-api" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115252 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b33eb0-d5c0-4160-8a3f-3c53d63f5a14" containerName="payload-pre-processing" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115257 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="15707df2-3b1c-43a8-8a99-b1f10231cb00" containerName="maas-api" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115263 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c775e5d-3134-40d3-8d81-4748781896a5" containerName="payload-processing" Oct 30 04:08:07.116858 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.115267 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" Oct 30 04:08:07.118652 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.118629 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.132745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.132723 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-dd81fd-openshift-default-dockercfg-g9rm4\"" Oct 30 04:08:07.147013 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.146991 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z"] Oct 30 04:08:07.188316 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188267 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-data\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188335 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd461436-cb68-4758-8592-62744af1dd53-istiod-ca-cert\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188367 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-credential-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188469 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-certs\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188493 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-istio-token\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188576 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd461436-cb68-4758-8592-62744af1dd53-istio-podinfo\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188620 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188722 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-envoy\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.188942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.188852 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtrcq\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-kube-api-access-xtrcq\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290389 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-certs\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290425 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-istio-token\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290458 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd461436-cb68-4758-8592-62744af1dd53-istio-podinfo\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290486 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290512 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-envoy\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290552 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtrcq\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-kube-api-access-xtrcq\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290600 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-data\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290642 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd461436-cb68-4758-8592-62744af1dd53-istiod-ca-cert\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.290674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290669 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-credential-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.291021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.290927 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-certs\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.291169 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.291144 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.291298 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.291213 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-data\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.291397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.291380 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd461436-cb68-4758-8592-62744af1dd53-istiod-ca-cert\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.291446 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.291424 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-credential-socket\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.293041 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.293016 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-envoy\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.293212 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.293193 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd461436-cb68-4758-8592-62744af1dd53-istio-podinfo\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.307626 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.307607 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-istio-token\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.307847 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.307829 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtrcq\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-kube-api-access-xtrcq\") pod \"e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.401254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.401174 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:07.432374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.432336 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:07.764180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.764154 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z"] Oct 30 04:08:07.766598 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:08:07.766568 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd461436_cb68_4758_8592_62744af1dd53.slice/crio-320cd1d588f426ce55057968e24155de5fd156d1f84b5169721f9bb6a798d4d4 WatchSource:0}: Error finding container 320cd1d588f426ce55057968e24155de5fd156d1f84b5169721f9bb6a798d4d4: Status 404 returned error can't find the container with id 320cd1d588f426ce55057968e24155de5fd156d1f84b5169721f9bb6a798d4d4 Oct 30 04:08:07.768957 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.768929 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:08:07.769043 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.768990 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:08:07.769043 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:07.769021 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:08:08.665290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:08.665255 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" event={"ID":"bd461436-cb68-4758-8592-62744af1dd53","Type":"ContainerStarted","Data":"ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315"} Oct 30 04:08:08.665726 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:08.665310 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" event={"ID":"bd461436-cb68-4758-8592-62744af1dd53","Type":"ContainerStarted","Data":"320cd1d588f426ce55057968e24155de5fd156d1f84b5169721f9bb6a798d4d4"} Oct 30 04:08:08.722232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:08.722179 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" podStartSLOduration=1.7221634080000001 podStartE2EDuration="1.722163408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:08:08.718525875 +0000 UTC m=+3912.720879264" watchObservedRunningTime="2026-10-30 04:08:08.722163408 +0000 UTC m=+3912.724516604" Oct 30 04:08:08.925233 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:08.925138 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:09.286868 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:09.286821 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" podUID="845e7f17-4711-4381-9302-7a3568a6e48d" containerName="payload-processing" containerID="cri-o://1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471" gracePeriod=25 Oct 30 04:08:09.336669 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:09.336631 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" podUID="193ae444-3be2-447a-a65e-90d467725718" containerName="payload-pre-processing" containerID="cri-o://edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639" gracePeriod=25 Oct 30 04:08:09.432768 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:09.432721 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:09.434503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:09.434475 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 04:08:09.434636 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:09.434526 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 04:08:10.112924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.112891 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8"] Oct 30 04:08:10.113447 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.113085 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" podUID="b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" containerName="maas-api" containerID="cri-o://2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338" gracePeriod=30 Oct 30 04:08:10.166509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.166479 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:10.433962 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.433869 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 04:08:10.434137 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.433964 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 04:08:10.529919 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.529883 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 04:08:10.646508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.646471 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-kube-api-access-pqj7q\" (UniqueName: \"kubernetes.io/projected/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-kube-api-access-pqj7q\") pod \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " Oct 30 04:08:10.646508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.646513 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-openshift-service-ca\") pod \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " Oct 30 04:08:10.646734 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.646559 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-maas-api-tls\") pod \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " Oct 30 04:08:10.646734 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.646588 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-metrics-certs\") pod \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\" (UID: \"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1\") " Oct 30 04:08:10.646943 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.646921 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-openshift-service-ca" pod "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" (UID: "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:10.648874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.648844 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-kube-api-access-pqj7q" pod "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" (UID: "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1"). InnerVolumeSpecName "kube-api-access-pqj7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:10.649017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.648875 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-metrics-certs" pod "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" (UID: "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:10.649017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.648896 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-maas-api-tls" pod "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" (UID: "b86d4db2-5c40-4ff1-a390-6dbf6c8204e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:10.680052 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.680023 4442 generic.go:356] "Generic (PLEG): container finished" podID="b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" containerID="2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338" exitCode=0 Oct 30 04:08:10.680218 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.680146 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" Oct 30 04:08:10.680218 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.680166 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" event={"ID":"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1","Type":"ContainerDied","Data":"2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338"} Oct 30 04:08:10.680366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.680268 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8" event={"ID":"b86d4db2-5c40-4ff1-a390-6dbf6c8204e1","Type":"ContainerDied","Data":"985b9c22c09b7f84514144a42708b693d00c94b9503ad95a694d6c938c50b677"} Oct 30 04:08:10.680366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.680303 4442 scope.go:122] "RemoveContainer" containerID="2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338" Oct 30 04:08:10.712507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.712487 4442 scope.go:122] "RemoveContainer" containerID="2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338" Oct 30 04:08:10.712860 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:10.712837 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338\": container with ID starting with 2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338 not found: ID does not exist" containerID="2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338" Oct 30 04:08:10.712956 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.712867 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338"} err="failed to get container status \"2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338\": rpc error: code = NotFound desc = could not find container \"2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338\": container with ID starting with 2f0a72074e71049c9d3aa63a032f092ceddb3c39ae9f5f9aee1725be41e41338 not found: ID does not exist" Oct 30 04:08:10.747766 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.747717 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqj7q\" (UniqueName: \"kubernetes.io/projected/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-kube-api-access-pqj7q\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:10.747880 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.747782 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:10.747880 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.747800 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:10.747880 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.747804 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8"] Oct 30 04:08:10.747880 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.747813 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:10.751110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.751089 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-852107-854bd5455f-zgtk8"] Oct 30 04:08:10.815639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:10.815609 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:11.433013 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.432972 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 04:08:11.433453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.433029 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 04:08:11.666342 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.666309 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx"] Oct 30 04:08:11.666561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.666536 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" containerID="cri-o://3a47a859f7569008eb7767ad7be062775ce5d7776a9d5a3169c2c96bdcc8a996" gracePeriod=30 Oct 30 04:08:11.677327 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.677294 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6"] Oct 30 04:08:11.677541 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.677517 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" containerID="cri-o://fa6e1030a84f5159fbdc2df76d4cefe13825ff4d7dd721799b6fa8f0e1708043" gracePeriod=30 Oct 30 04:08:11.679702 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.679677 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6"] Oct 30 04:08:11.679902 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:11.679881 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" containerID="cri-o://583711099fc931555768c78b9656755b56fdbbdc1b45a088ee2beb35afa7ade9" gracePeriod=30 Oct 30 04:08:12.119353 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:12.119314 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" path="/var/lib/kubelet/pods/b86d4db2-5c40-4ff1-a390-6dbf6c8204e1/volumes" Oct 30 04:08:12.334991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:12.334960 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:12.437794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:12.437290 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 04:08:12.437794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:12.437359 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 04:08:12.437794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:12.437389 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:12.533546 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:12.533494 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" podUID="193ae444-3be2-447a-a65e-90d467725718" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:08:12.533729 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:12.533563 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" podUID="845e7f17-4711-4381-9302-7a3568a6e48d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:08:13.061815 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.061743 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:13.344996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.344886 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:13.434783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.433190 4442 patch_prober.go:29] interesting pod/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Oct 30 04:08:13.434783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.433272 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Oct 30 04:08:13.507780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.507711 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:13.614294 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.614215 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:13.716347 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.716314 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:13.970362 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:13.970278 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:14.118504 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:14.118475 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:14.584115 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:14.584081 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:14.584554 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:14.584460 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:14.585442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:14.585423 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:08:15.716142 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:15.716109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:16.811064 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.811028 4442 generic.go:356] "Generic (PLEG): container finished" podID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerID="3a47a859f7569008eb7767ad7be062775ce5d7776a9d5a3169c2c96bdcc8a996" exitCode=0 Oct 30 04:08:16.811423 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.811110 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" event={"ID":"3f7e4e5c-5372-4cea-aa2f-8d63671314de","Type":"ContainerDied","Data":"3a47a859f7569008eb7767ad7be062775ce5d7776a9d5a3169c2c96bdcc8a996"} Oct 30 04:08:16.813128 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.813103 4442 generic.go:356] "Generic (PLEG): container finished" podID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerID="fa6e1030a84f5159fbdc2df76d4cefe13825ff4d7dd721799b6fa8f0e1708043" exitCode=0 Oct 30 04:08:16.813245 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.813175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" event={"ID":"f46ebcfb-6be1-47de-95a2-4b8876205c34","Type":"ContainerDied","Data":"fa6e1030a84f5159fbdc2df76d4cefe13825ff4d7dd721799b6fa8f0e1708043"} Oct 30 04:08:16.814975 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.814956 4442 generic.go:356] "Generic (PLEG): container finished" podID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerID="583711099fc931555768c78b9656755b56fdbbdc1b45a088ee2beb35afa7ade9" exitCode=0 Oct 30 04:08:16.815058 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.815002 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" event={"ID":"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68","Type":"ContainerDied","Data":"583711099fc931555768c78b9656755b56fdbbdc1b45a088ee2beb35afa7ade9"} Oct 30 04:08:16.968446 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.968421 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 04:08:16.989427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:16.982004 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 04:08:17.022556 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.022525 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 04:08:17.142084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142053 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-token\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.142084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142093 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istiod-ca-cert\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142114 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-envoy\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142144 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-podinfo\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142176 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-data\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142207 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-socket\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142241 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-certs\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142273 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ebcfb-6be1-47de-95a2-4b8876205c34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ebcfb-6be1-47de-95a2-4b8876205c34-istiod-ca-cert\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142312 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-certs\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.142345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142347 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-kube-api-access-2lqd5\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-kube-api-access-2lqd5\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142375 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-podinfo\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142407 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-envoy\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142436 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istiod-ca-cert\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142482 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-credential-socket\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142509 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-token\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-token\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142559 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-certs\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142594 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-kube-api-access-vzdlx\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-kube-api-access-vzdlx\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142633 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-socket\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142668 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-socket\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142712 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-token\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-token\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142770 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-envoy\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142828 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-credential-socket\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142833 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-certs" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142949 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-data\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.142991 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-data\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143103 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-credential-socket" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.143110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143112 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-podinfo\") pod \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\" (UID: \"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68\") " Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143153 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-credential-socket\") pod \"f46ebcfb-6be1-47de-95a2-4b8876205c34\" (UID: \"f46ebcfb-6be1-47de-95a2-4b8876205c34\") " Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143202 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-kube-api-access-rw4nr\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-kube-api-access-rw4nr\") pod \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\" (UID: \"3f7e4e5c-5372-4cea-aa2f-8d63671314de\") " Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143297 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-data" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143409 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istiod-ca-cert" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143480 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-data" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143554 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istiod-ca-cert" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143638 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-credential-socket" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143923 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143947 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143959 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143973 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.143994 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144010 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144011 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-certs" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.144186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144023 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.145005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144217 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-socket" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.145005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ebcfb-6be1-47de-95a2-4b8876205c34-istiod-ca-cert" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:17.145005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144578 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-data" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.145005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144645 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-credential-socket" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.145005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.144802 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-socket" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.145278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.145006 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-certs" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.145691 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.145629 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-podinfo" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:08:17.146456 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.146360 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-socket" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.146802 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.146738 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-envoy" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.147486 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.147366 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-token" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:17.147486 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.147454 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-envoy" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.147486 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.147459 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-kube-api-access-rw4nr" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "kube-api-access-rw4nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:17.147971 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.147946 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-token" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:17.148214 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.148183 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-token" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:17.148659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.148635 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-kube-api-access-vzdlx" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "kube-api-access-vzdlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:17.148950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.148925 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-podinfo" pod "f46ebcfb-6be1-47de-95a2-4b8876205c34" (UID: "f46ebcfb-6be1-47de-95a2-4b8876205c34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:08:17.149090 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.149069 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-envoy" pod "3f7e4e5c-5372-4cea-aa2f-8d63671314de" (UID: "3f7e4e5c-5372-4cea-aa2f-8d63671314de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:17.149177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.149164 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-kube-api-access-2lqd5" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "kube-api-access-2lqd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:17.149602 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.149578 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-podinfo" pod "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" (UID: "2f9cbdcf-886d-46e5-a7cd-7321e55e1a68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245580 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245605 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245614 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245623 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdlx\" (UniqueName: \"kubernetes.io/projected/f46ebcfb-6be1-47de-95a2-4b8876205c34-kube-api-access-vzdlx\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245632 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245641 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245649 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245657 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.245659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245666 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245673 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245681 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ebcfb-6be1-47de-95a2-4b8876205c34-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245690 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4nr\" (UniqueName: \"kubernetes.io/projected/3f7e4e5c-5372-4cea-aa2f-8d63671314de-kube-api-access-rw4nr\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245698 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245706 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245717 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f7e4e5c-5372-4cea-aa2f-8d63671314de-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245725 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245733 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ebcfb-6be1-47de-95a2-4b8876205c34-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245740 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f7e4e5c-5372-4cea-aa2f-8d63671314de-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245763 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lqd5\" (UniqueName: \"kubernetes.io/projected/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68-kube-api-access-2lqd5\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.246180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.245774 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ebcfb-6be1-47de-95a2-4b8876205c34-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:17.294444 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.294415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:17.822493 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.822462 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" Oct 30 04:08:17.822493 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.822476 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx" event={"ID":"3f7e4e5c-5372-4cea-aa2f-8d63671314de","Type":"ContainerDied","Data":"bbd69ef800464efc8c2a74ec48bfef9f5251419c64e78e457dee2a37a730ebc0"} Oct 30 04:08:17.823053 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.822534 4442 scope.go:122] "RemoveContainer" containerID="3a47a859f7569008eb7767ad7be062775ce5d7776a9d5a3169c2c96bdcc8a996" Oct 30 04:08:17.824166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.824145 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" Oct 30 04:08:17.824259 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.824155 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6" event={"ID":"f46ebcfb-6be1-47de-95a2-4b8876205c34","Type":"ContainerDied","Data":"d62b5670ca6e04cfddeff79bcd613f8a0a3d4043c55465c2f328dcb88e982aee"} Oct 30 04:08:17.826055 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.825944 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" event={"ID":"2f9cbdcf-886d-46e5-a7cd-7321e55e1a68","Type":"ContainerDied","Data":"91c4329b113f48e5434d309c4795a315c7daf1a50dadb0adaa7337fa6feb2036"} Oct 30 04:08:17.826055 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.825990 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6" Oct 30 04:08:17.833910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.833886 4442 scope.go:122] "RemoveContainer" containerID="fa6e1030a84f5159fbdc2df76d4cefe13825ff4d7dd721799b6fa8f0e1708043" Oct 30 04:08:17.843744 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.843727 4442 scope.go:122] "RemoveContainer" containerID="583711099fc931555768c78b9656755b56fdbbdc1b45a088ee2beb35afa7ade9" Oct 30 04:08:17.895119 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.895096 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6"] Oct 30 04:08:17.905820 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.905793 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-2kwq6"] Oct 30 04:08:17.935832 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.935807 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx"] Oct 30 04:08:17.939501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.939474 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-28ksx"] Oct 30 04:08:17.969331 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.969271 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6"] Oct 30 04:08:17.973122 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:17.973101 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-55vj6"] Oct 30 04:08:18.106725 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:18.106697 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" path="/var/lib/kubelet/pods/2f9cbdcf-886d-46e5-a7cd-7321e55e1a68/volumes" Oct 30 04:08:18.107168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:18.107155 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" path="/var/lib/kubelet/pods/3f7e4e5c-5372-4cea-aa2f-8d63671314de/volumes" Oct 30 04:08:18.107558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:18.107539 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" path="/var/lib/kubelet/pods/f46ebcfb-6be1-47de-95a2-4b8876205c34/volumes" Oct 30 04:08:18.807332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:18.807299 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:18.962074 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:18.962033 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc"] Oct 30 04:08:19.059247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:19.059166 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:19.668682 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:19.668646 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:20.131038 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:20.131002 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:20.403010 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:20.402927 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:21.794980 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:21.794934 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:22.533213 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:22.533176 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" podUID="193ae444-3be2-447a-a65e-90d467725718" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:08:22.533381 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:22.533256 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" podUID="845e7f17-4711-4381-9302-7a3568a6e48d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:08:23.295454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:23.295411 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:23.962982 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:23.962933 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:08:24.811862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:24.811817 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:24.860572 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:24.860507 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:08:25.439814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:25.439773 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:25.543424 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:25.542468 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:26.486145 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:26.486109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:27.460955 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:27.460923 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:27.749250 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:27.749213 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:27.906424 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:27.906389 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:29.060661 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.060628 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4"] Oct 30 04:08:29.061319 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.060874 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" podUID="5c36d571-4774-49b7-8e1e-25821eb776d5" containerName="maas-api" containerID="cri-o://9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047" gracePeriod=30 Oct 30 04:08:29.119016 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.118987 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:29.448166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.448144 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 04:08:29.461737 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.461706 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-metrics-certs\") pod \"5c36d571-4774-49b7-8e1e-25821eb776d5\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " Oct 30 04:08:29.461943 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.461924 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c36d571-4774-49b7-8e1e-25821eb776d5-kube-api-access-57bjb\" (UniqueName: \"kubernetes.io/projected/5c36d571-4774-49b7-8e1e-25821eb776d5-kube-api-access-57bjb\") pod \"5c36d571-4774-49b7-8e1e-25821eb776d5\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " Oct 30 04:08:29.462050 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.461986 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls\") pod \"5c36d571-4774-49b7-8e1e-25821eb776d5\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " Oct 30 04:08:29.462050 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.462028 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c36d571-4774-49b7-8e1e-25821eb776d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c36d571-4774-49b7-8e1e-25821eb776d5-openshift-service-ca\") pod \"5c36d571-4774-49b7-8e1e-25821eb776d5\" (UID: \"5c36d571-4774-49b7-8e1e-25821eb776d5\") " Oct 30 04:08:29.462614 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.462590 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c36d571-4774-49b7-8e1e-25821eb776d5-openshift-service-ca" pod "5c36d571-4774-49b7-8e1e-25821eb776d5" (UID: "5c36d571-4774-49b7-8e1e-25821eb776d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:29.467191 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.467162 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-metrics-certs" pod "5c36d571-4774-49b7-8e1e-25821eb776d5" (UID: "5c36d571-4774-49b7-8e1e-25821eb776d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:29.467308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.467290 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c36d571-4774-49b7-8e1e-25821eb776d5-kube-api-access-57bjb" pod "5c36d571-4774-49b7-8e1e-25821eb776d5" (UID: "5c36d571-4774-49b7-8e1e-25821eb776d5"). InnerVolumeSpecName "kube-api-access-57bjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:29.474148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.474123 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls" pod "5c36d571-4774-49b7-8e1e-25821eb776d5" (UID: "5c36d571-4774-49b7-8e1e-25821eb776d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:29.562950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.562884 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57bjb\" (UniqueName: \"kubernetes.io/projected/5c36d571-4774-49b7-8e1e-25821eb776d5-kube-api-access-57bjb\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.562950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.562915 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.562950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.562926 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c36d571-4774-49b7-8e1e-25821eb776d5-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.562950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.562935 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c36d571-4774-49b7-8e1e-25821eb776d5-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.625592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.625566 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 04:08:29.635054 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.635031 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 04:08:29.663771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.663729 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845e7f17-4711-4381-9302-7a3568a6e48d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e7f17-4711-4381-9302-7a3568a6e48d-plugins-config-volume\") pod \"845e7f17-4711-4381-9302-7a3568a6e48d\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " Oct 30 04:08:29.663922 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.663822 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193ae444-3be2-447a-a65e-90d467725718-kube-api-access-nrz6n\" (UniqueName: \"kubernetes.io/projected/193ae444-3be2-447a-a65e-90d467725718-kube-api-access-nrz6n\") pod \"193ae444-3be2-447a-a65e-90d467725718\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " Oct 30 04:08:29.663922 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.663883 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193ae444-3be2-447a-a65e-90d467725718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ae444-3be2-447a-a65e-90d467725718-plugins-config-volume\") pod \"193ae444-3be2-447a-a65e-90d467725718\" (UID: \"193ae444-3be2-447a-a65e-90d467725718\") " Oct 30 04:08:29.664074 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.664055 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845e7f17-4711-4381-9302-7a3568a6e48d-kube-api-access-xl9qp\" (UniqueName: \"kubernetes.io/projected/845e7f17-4711-4381-9302-7a3568a6e48d-kube-api-access-xl9qp\") pod \"845e7f17-4711-4381-9302-7a3568a6e48d\" (UID: \"845e7f17-4711-4381-9302-7a3568a6e48d\") " Oct 30 04:08:29.664984 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.664955 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193ae444-3be2-447a-a65e-90d467725718-plugins-config-volume" pod "193ae444-3be2-447a-a65e-90d467725718" (UID: "193ae444-3be2-447a-a65e-90d467725718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:29.665129 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.665094 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845e7f17-4711-4381-9302-7a3568a6e48d-plugins-config-volume" pod "845e7f17-4711-4381-9302-7a3568a6e48d" (UID: "845e7f17-4711-4381-9302-7a3568a6e48d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:29.666685 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.666661 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193ae444-3be2-447a-a65e-90d467725718-kube-api-access-nrz6n" pod "193ae444-3be2-447a-a65e-90d467725718" (UID: "193ae444-3be2-447a-a65e-90d467725718"). InnerVolumeSpecName "kube-api-access-nrz6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:29.666837 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.666735 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845e7f17-4711-4381-9302-7a3568a6e48d-kube-api-access-xl9qp" pod "845e7f17-4711-4381-9302-7a3568a6e48d" (UID: "845e7f17-4711-4381-9302-7a3568a6e48d"). InnerVolumeSpecName "kube-api-access-xl9qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:29.722585 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.722549 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:29.765918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.765804 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl9qp\" (UniqueName: \"kubernetes.io/projected/845e7f17-4711-4381-9302-7a3568a6e48d-kube-api-access-xl9qp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.765918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.765840 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/845e7f17-4711-4381-9302-7a3568a6e48d-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.765918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.765856 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrz6n\" (UniqueName: \"kubernetes.io/projected/193ae444-3be2-447a-a65e-90d467725718-kube-api-access-nrz6n\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.765918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.765869 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193ae444-3be2-447a-a65e-90d467725718-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:29.905510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.905419 4442 generic.go:356] "Generic (PLEG): container finished" podID="5c36d571-4774-49b7-8e1e-25821eb776d5" containerID="9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047" exitCode=0 Oct 30 04:08:29.905510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.905466 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" Oct 30 04:08:29.905732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.905510 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" event={"ID":"5c36d571-4774-49b7-8e1e-25821eb776d5","Type":"ContainerDied","Data":"9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047"} Oct 30 04:08:29.905732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.905575 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4" event={"ID":"5c36d571-4774-49b7-8e1e-25821eb776d5","Type":"ContainerDied","Data":"f8422dc16d98798052cab53d4b5fb808259f73678e5c60dc6dab45433fc8a536"} Oct 30 04:08:29.905732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.905608 4442 scope.go:122] "RemoveContainer" containerID="9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047" Oct 30 04:08:29.907066 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.907047 4442 generic.go:356] "Generic (PLEG): container finished" podID="193ae444-3be2-447a-a65e-90d467725718" containerID="edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639" exitCode=0 Oct 30 04:08:29.907174 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.907096 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" Oct 30 04:08:29.907174 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.907128 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" event={"ID":"193ae444-3be2-447a-a65e-90d467725718","Type":"ContainerDied","Data":"edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639"} Oct 30 04:08:29.907274 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.907175 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6" event={"ID":"193ae444-3be2-447a-a65e-90d467725718","Type":"ContainerDied","Data":"f1c6f5a1d2d2a8dc4488c1f0299d8475e627570eceb1a820092b4d18002e9957"} Oct 30 04:08:29.908804 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.908781 4442 generic.go:356] "Generic (PLEG): container finished" podID="845e7f17-4711-4381-9302-7a3568a6e48d" containerID="1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471" exitCode=0 Oct 30 04:08:29.908930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.908816 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" Oct 30 04:08:29.908930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.908871 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" event={"ID":"845e7f17-4711-4381-9302-7a3568a6e48d","Type":"ContainerDied","Data":"1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471"} Oct 30 04:08:29.909048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.908937 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p" event={"ID":"845e7f17-4711-4381-9302-7a3568a6e48d","Type":"ContainerDied","Data":"1eb22f41274cb6a414bffb1cdf3fabb20f2fe0affed3734925eceb472f7a2aec"} Oct 30 04:08:29.915532 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.915509 4442 scope.go:122] "RemoveContainer" containerID="9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047" Oct 30 04:08:29.915839 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:29.915813 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047\": container with ID starting with 9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047 not found: ID does not exist" containerID="9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047" Oct 30 04:08:29.915958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.915845 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047"} err="failed to get container status \"9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047\": rpc error: code = NotFound desc = could not find container \"9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047\": container with ID starting with 9e0e5da69d484e0bd33235f41c430727b853fd596c0bc2ec367b13a425418047 not found: ID does not exist" Oct 30 04:08:29.915958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.915863 4442 scope.go:122] "RemoveContainer" containerID="edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639" Oct 30 04:08:29.926074 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.926055 4442 scope.go:122] "RemoveContainer" containerID="edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639" Oct 30 04:08:29.926354 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:29.926335 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639\": container with ID starting with edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639 not found: ID does not exist" containerID="edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639" Oct 30 04:08:29.926419 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.926361 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639"} err="failed to get container status \"edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639\": rpc error: code = NotFound desc = could not find container \"edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639\": container with ID starting with edf216bb78d655285c32daed29df1afa6fa76290cb586c61a6d170b80b4f0639 not found: ID does not exist" Oct 30 04:08:29.926419 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.926376 4442 scope.go:122] "RemoveContainer" containerID="1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471" Oct 30 04:08:29.934660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.934640 4442 scope.go:122] "RemoveContainer" containerID="1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471" Oct 30 04:08:29.934950 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:29.934926 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471\": container with ID starting with 1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471 not found: ID does not exist" containerID="1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471" Oct 30 04:08:29.935038 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.934956 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471"} err="failed to get container status \"1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471\": rpc error: code = NotFound desc = could not find container \"1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471\": container with ID starting with 1c570e8d55bbc0886a8bb09c3e29e5ca960518e078e2a73c0a874a984b1cd471 not found: ID does not exist" Oct 30 04:08:29.980926 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.980892 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p"] Oct 30 04:08:29.985643 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:29.985606 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-852107-5d944d8796-w5d9p"] Oct 30 04:08:30.013471 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:30.013432 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6"] Oct 30 04:08:30.016036 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:30.016011 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-852107-7bd96c6c79-vgqg6"] Oct 30 04:08:30.045552 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:30.045527 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4"] Oct 30 04:08:30.048822 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:30.048799 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3c874f-865cb5cb9d-lfvs4"] Oct 30 04:08:30.105698 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:30.105673 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193ae444-3be2-447a-a65e-90d467725718" path="/var/lib/kubelet/pods/193ae444-3be2-447a-a65e-90d467725718/volumes" Oct 30 04:08:30.106136 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:30.106121 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c36d571-4774-49b7-8e1e-25821eb776d5" path="/var/lib/kubelet/pods/5c36d571-4774-49b7-8e1e-25821eb776d5/volumes" Oct 30 04:08:30.106464 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:30.106452 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845e7f17-4711-4381-9302-7a3568a6e48d" path="/var/lib/kubelet/pods/845e7f17-4711-4381-9302-7a3568a6e48d/volumes" Oct 30 04:08:31.270736 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:31.270702 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:32.674545 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:32.674504 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:33.771573 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:33.771538 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:33.983363 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:33.983322 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" podUID="d5452058-93bf-4de6-a642-2dedbad09bd2" containerName="main" containerID="cri-o://9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4" gracePeriod=45 Oct 30 04:08:33.987658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:33.987625 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" podUID="d5452058-93bf-4de6-a642-2dedbad09bd2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": EOF" Oct 30 04:08:34.060930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.060866 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:34.184549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.184522 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:34.255969 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.255945 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 04:08:34.311629 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311542 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5452058-93bf-4de6-a642-2dedbad09bd2-kube-api-access-xxt66\" (UniqueName: \"kubernetes.io/projected/d5452058-93bf-4de6-a642-2dedbad09bd2-kube-api-access-xxt66\") pod \"d5452058-93bf-4de6-a642-2dedbad09bd2\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " Oct 30 04:08:34.311629 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311616 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-home\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-home\") pod \"d5452058-93bf-4de6-a642-2dedbad09bd2\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " Oct 30 04:08:34.311934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311741 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-tmp-dir\") pod \"d5452058-93bf-4de6-a642-2dedbad09bd2\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " Oct 30 04:08:34.311934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311842 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-dshm\") pod \"d5452058-93bf-4de6-a642-2dedbad09bd2\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " Oct 30 04:08:34.311934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311872 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5452058-93bf-4de6-a642-2dedbad09bd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5452058-93bf-4de6-a642-2dedbad09bd2-tls-certs\") pod \"d5452058-93bf-4de6-a642-2dedbad09bd2\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " Oct 30 04:08:34.311934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311897 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-model-cache\") pod \"d5452058-93bf-4de6-a642-2dedbad09bd2\" (UID: \"d5452058-93bf-4de6-a642-2dedbad09bd2\") " Oct 30 04:08:34.312204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311963 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-home" pod "d5452058-93bf-4de6-a642-2dedbad09bd2" (UID: "d5452058-93bf-4de6-a642-2dedbad09bd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.312204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.311981 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-tmp-dir" pod "d5452058-93bf-4de6-a642-2dedbad09bd2" (UID: "d5452058-93bf-4de6-a642-2dedbad09bd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.312326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.312242 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-model-cache" pod "d5452058-93bf-4de6-a642-2dedbad09bd2" (UID: "d5452058-93bf-4de6-a642-2dedbad09bd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.312387 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.312339 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.312387 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.312359 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.314608 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.314577 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5452058-93bf-4de6-a642-2dedbad09bd2-tls-certs" pod "d5452058-93bf-4de6-a642-2dedbad09bd2" (UID: "d5452058-93bf-4de6-a642-2dedbad09bd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:34.314735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.314721 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-dshm" pod "d5452058-93bf-4de6-a642-2dedbad09bd2" (UID: "d5452058-93bf-4de6-a642-2dedbad09bd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:34.314867 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.314837 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5452058-93bf-4de6-a642-2dedbad09bd2-kube-api-access-xxt66" pod "d5452058-93bf-4de6-a642-2dedbad09bd2" (UID: "d5452058-93bf-4de6-a642-2dedbad09bd2"). InnerVolumeSpecName "kube-api-access-xxt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:34.413352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.413317 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.413352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.413346 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5452058-93bf-4de6-a642-2dedbad09bd2-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.413352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.413356 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5452058-93bf-4de6-a642-2dedbad09bd2-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.413352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.413364 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxt66\" (UniqueName: \"kubernetes.io/projected/d5452058-93bf-4de6-a642-2dedbad09bd2-kube-api-access-xxt66\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:34.713778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.713669 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:34.939242 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.939199 4442 generic.go:356] "Generic (PLEG): container finished" podID="d5452058-93bf-4de6-a642-2dedbad09bd2" containerID="9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4" exitCode=0 Oct 30 04:08:34.939713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.939250 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" Oct 30 04:08:34.939713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.939281 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" event={"ID":"d5452058-93bf-4de6-a642-2dedbad09bd2","Type":"ContainerDied","Data":"9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4"} Oct 30 04:08:34.939713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.939328 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc" event={"ID":"d5452058-93bf-4de6-a642-2dedbad09bd2","Type":"ContainerDied","Data":"4bc5b8269ad3b8fa95dfbd2b3d37a4ca5ea6e3c4b83f9304c29c5f37819ef2b1"} Oct 30 04:08:34.939713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.939345 4442 scope.go:122] "RemoveContainer" containerID="9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4" Oct 30 04:08:34.951271 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.951255 4442 scope.go:122] "RemoveContainer" containerID="9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4" Oct 30 04:08:34.951518 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:34.951500 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4\": container with ID starting with 9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4 not found: ID does not exist" containerID="9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4" Oct 30 04:08:34.951579 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.951524 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4"} err="failed to get container status \"9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4\": rpc error: code = NotFound desc = could not find container \"9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4\": container with ID starting with 9a742f7a243f702eac2859608d60ba0c571e7bbe5eaa473a3ab077ae5eb98bf4 not found: ID does not exist" Oct 30 04:08:34.991816 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.986217 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc"] Oct 30 04:08:34.996006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:34.995973 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d80b1/premium-w4-8d80b1-kserve-7f8c66fb8f-rr4zc"] Oct 30 04:08:35.116810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:35.115106 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p"] Oct 30 04:08:35.116810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:35.115363 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" containerID="cri-o://81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791" gracePeriod=30 Oct 30 04:08:35.149699 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:35.149636 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:35.892851 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:35.892817 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:36.105497 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:36.105459 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5452058-93bf-4de6-a642-2dedbad09bd2" path="/var/lib/kubelet/pods/d5452058-93bf-4de6-a642-2dedbad09bd2/volumes" Oct 30 04:08:37.481334 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:37.481302 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:38.409805 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:38.409774 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:38.509414 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:38.509384 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:39.083518 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:39.083488 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:39.661566 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:39.661534 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:40.374730 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.374688 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 04:08:40.472521 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472438 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-certs\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.472521 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472500 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107b2e13-5fb2-4241-8143-e75258d44261-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107b2e13-5fb2-4241-8143-e75258d44261-istiod-ca-cert\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.472779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472528 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-socket\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.472779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472574 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-credential-socket\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.472779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472600 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-data\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.472779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472634 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107b2e13-5fb2-4241-8143-e75258d44261-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107b2e13-5fb2-4241-8143-e75258d44261-istio-podinfo\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.472779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472720 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-envoy\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.472779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472742 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-istio-token\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-istio-token\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.473107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472783 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-kube-api-access-n84d4\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-kube-api-access-n84d4\") pod \"107b2e13-5fb2-4241-8143-e75258d44261\" (UID: \"107b2e13-5fb2-4241-8143-e75258d44261\") " Oct 30 04:08:40.473107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472778 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-certs" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:40.473107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472907 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-credential-socket" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:40.473107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.472929 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-socket" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:40.473107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.473007 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-data" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:40.473107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.473062 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107b2e13-5fb2-4241-8143-e75258d44261-istiod-ca-cert" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:40.473397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.473176 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.473397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.473190 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.473397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.473199 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.473397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.473207 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107b2e13-5fb2-4241-8143-e75258d44261-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.473397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.473214 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.475095 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.475070 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-envoy" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:08:40.475290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.475270 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-istio-token" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:40.475561 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.475539 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107b2e13-5fb2-4241-8143-e75258d44261-istio-podinfo" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:08:40.475660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.475586 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-kube-api-access-n84d4" pod "107b2e13-5fb2-4241-8143-e75258d44261" (UID: "107b2e13-5fb2-4241-8143-e75258d44261"). InnerVolumeSpecName "kube-api-access-n84d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:40.487817 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.487791 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:40.574287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.574251 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107b2e13-5fb2-4241-8143-e75258d44261-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.574287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.574288 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107b2e13-5fb2-4241-8143-e75258d44261-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.574514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.574301 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.574514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.574313 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84d4\" (UniqueName: \"kubernetes.io/projected/107b2e13-5fb2-4241-8143-e75258d44261-kube-api-access-n84d4\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:40.974746 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.974713 4442 generic.go:356] "Generic (PLEG): container finished" podID="107b2e13-5fb2-4241-8143-e75258d44261" containerID="81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791" exitCode=0 Oct 30 04:08:40.975175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.974775 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" event={"ID":"107b2e13-5fb2-4241-8143-e75258d44261","Type":"ContainerDied","Data":"81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791"} Oct 30 04:08:40.975175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.974800 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" Oct 30 04:08:40.975175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.974850 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p" event={"ID":"107b2e13-5fb2-4241-8143-e75258d44261","Type":"ContainerDied","Data":"e0c4dc6bcc1e31411070ded8b46166b3aa3a9968eae4d340d24d26f31e00fb10"} Oct 30 04:08:40.975175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.974873 4442 scope.go:122] "RemoveContainer" containerID="81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791" Oct 30 04:08:40.985263 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.985235 4442 scope.go:122] "RemoveContainer" containerID="d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16" Oct 30 04:08:40.995886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.995866 4442 scope.go:122] "RemoveContainer" containerID="81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791" Oct 30 04:08:40.996300 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:40.996275 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791\": container with ID starting with 81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791 not found: ID does not exist" containerID="81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791" Oct 30 04:08:40.996429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.996310 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791"} err="failed to get container status \"81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791\": rpc error: code = NotFound desc = could not find container \"81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791\": container with ID starting with 81a975bdc2e670187a375b2486a352978bd2281e397301e6035fba2e0db66791 not found: ID does not exist" Oct 30 04:08:40.996429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.996336 4442 scope.go:122] "RemoveContainer" containerID="d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16" Oct 30 04:08:40.996684 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:40.996658 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16\": container with ID starting with d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16 not found: ID does not exist" containerID="d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16" Oct 30 04:08:40.996847 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:40.996693 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16"} err="failed to get container status \"d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16\": rpc error: code = NotFound desc = could not find container \"d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16\": container with ID starting with d37f78d5cdb2ed4cc51a6eff19464fbb0e314f4c9e15373e3e433dcecc370c16 not found: ID does not exist" Oct 30 04:08:41.020169 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.020145 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p"] Oct 30 04:08:41.031133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.031107 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3c874f-openshift-default-75f5654dfc-mx95p"] Oct 30 04:08:41.649899 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.649862 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9"] Oct 30 04:08:41.650126 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.650098 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" podUID="14b80988-fbb8-4d18-910b-190fc3c6bcdd" containerName="maas-api" containerID="cri-o://627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874" gracePeriod=30 Oct 30 04:08:41.701411 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.701386 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:41.925701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.925673 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 04:08:41.989123 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.989085 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-metrics-certs\") pod \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " Oct 30 04:08:41.989650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.989144 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b80988-fbb8-4d18-910b-190fc3c6bcdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b80988-fbb8-4d18-910b-190fc3c6bcdd-openshift-service-ca\") pod \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " Oct 30 04:08:41.989650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.989262 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b80988-fbb8-4d18-910b-190fc3c6bcdd-kube-api-access-68dwv\" (UniqueName: \"kubernetes.io/projected/14b80988-fbb8-4d18-910b-190fc3c6bcdd-kube-api-access-68dwv\") pod \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " Oct 30 04:08:41.989650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.989339 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-maas-api-tls\") pod \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\" (UID: \"14b80988-fbb8-4d18-910b-190fc3c6bcdd\") " Oct 30 04:08:41.990161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.990128 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b80988-fbb8-4d18-910b-190fc3c6bcdd-openshift-service-ca" pod "14b80988-fbb8-4d18-910b-190fc3c6bcdd" (UID: "14b80988-fbb8-4d18-910b-190fc3c6bcdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:08:41.992709 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.992671 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-maas-api-tls" pod "14b80988-fbb8-4d18-910b-190fc3c6bcdd" (UID: "14b80988-fbb8-4d18-910b-190fc3c6bcdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:41.996940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.996682 4442 generic.go:356] "Generic (PLEG): container finished" podID="14b80988-fbb8-4d18-910b-190fc3c6bcdd" containerID="627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874" exitCode=0 Oct 30 04:08:41.997492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.997355 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" event={"ID":"14b80988-fbb8-4d18-910b-190fc3c6bcdd","Type":"ContainerDied","Data":"627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874"} Oct 30 04:08:41.997492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.997417 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" event={"ID":"14b80988-fbb8-4d18-910b-190fc3c6bcdd","Type":"ContainerDied","Data":"7b561d51d9fdd96d403cff1bb589ed9de685801ed4e3d6ff560de839458e55d3"} Oct 30 04:08:41.997492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.997447 4442 scope.go:122] "RemoveContainer" containerID="627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874" Oct 30 04:08:41.998633 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:41.997866 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9" Oct 30 04:08:42.020025 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.014144 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b80988-fbb8-4d18-910b-190fc3c6bcdd-kube-api-access-68dwv" pod "14b80988-fbb8-4d18-910b-190fc3c6bcdd" (UID: "14b80988-fbb8-4d18-910b-190fc3c6bcdd"). InnerVolumeSpecName "kube-api-access-68dwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:08:42.029942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.029902 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-metrics-certs" pod "14b80988-fbb8-4d18-910b-190fc3c6bcdd" (UID: "14b80988-fbb8-4d18-910b-190fc3c6bcdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:08:42.030935 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.030907 4442 scope.go:122] "RemoveContainer" containerID="627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874" Oct 30 04:08:42.031341 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:42.031317 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874\": container with ID starting with 627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874 not found: ID does not exist" containerID="627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874" Oct 30 04:08:42.031413 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.031354 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874"} err="failed to get container status \"627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874\": rpc error: code = NotFound desc = could not find container \"627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874\": container with ID starting with 627139eddd834f0a69cc5ebeb3bda715a394cfce94939f02e16d91b9d4dac874 not found: ID does not exist" Oct 30 04:08:42.091204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.091173 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:42.091204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.091206 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14b80988-fbb8-4d18-910b-190fc3c6bcdd-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:42.091402 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.091219 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68dwv\" (UniqueName: \"kubernetes.io/projected/14b80988-fbb8-4d18-910b-190fc3c6bcdd-kube-api-access-68dwv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:42.091402 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.091233 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14b80988-fbb8-4d18-910b-190fc3c6bcdd-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:08:42.105996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.105972 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107b2e13-5fb2-4241-8143-e75258d44261" path="/var/lib/kubelet/pods/107b2e13-5fb2-4241-8143-e75258d44261/volumes" Oct 30 04:08:42.309630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.309606 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:42.332824 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.332796 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9"] Oct 30 04:08:42.352071 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:42.352033 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d80b1-8db4646cc-x99r9"] Oct 30 04:08:43.700568 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:43.700534 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:44.053219 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:44.053182 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:08:44.106172 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:44.106145 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b80988-fbb8-4d18-910b-190fc3c6bcdd" path="/var/lib/kubelet/pods/14b80988-fbb8-4d18-910b-190fc3c6bcdd/volumes" Oct 30 04:08:45.365833 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:45.365799 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:46.712215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:46.712173 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:46.812343 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:46.812315 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:46.917016 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:46.916985 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:47.138101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:47.138072 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:47.273847 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:47.273793 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:47.370933 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:47.370900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:47.470171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:47.470104 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:47.938817 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:47.938776 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:48.409836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:48.409807 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:48.504365 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:48.502823 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:49.990828 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:49.990790 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:51.487747 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:51.487717 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:52.683492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:52.683440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:53.030706 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:53.030673 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:53.568770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:53.568723 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:53.687442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:53.687407 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:54.495022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:54.494987 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:54.860533 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:08:54.860500 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:08:55.452593 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:55.452560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:55.595516 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:55.595488 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:55.830156 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:55.830121 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:55.989792 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:55.989734 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:56.090173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:56.090080 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:57.471071 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:57.471040 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:57.570956 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:57.570921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:58.902240 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:08:58.902206 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:00.232337 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:00.232303 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:09:00.313903 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:00.313860 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:01.802594 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:01.802557 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:02.934985 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:02.934952 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:03.205283 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:03.205205 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:04.695650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:04.695621 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:06.201011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:06.200972 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:07.895741 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:07.895708 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:09.308620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:09.308587 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:10.795071 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:10.795031 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:12.308372 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:12.308331 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:13.705977 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:13.705939 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:13.802990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:13.802956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:15.200092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:15.200060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:16.694127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:16.694093 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:18.088849 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:18.088814 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:19.582773 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:19.582721 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:19.660588 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:19.660547 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:09:21.107558 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:21.107530 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:22.497369 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:22.497334 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:23.891066 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:23.891030 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:24.861339 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:09:24.861307 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:09:25.292128 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:25.292097 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:26.789658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:26.789623 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:27.781530 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:27.781486 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:28.344601 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:28.344570 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:29.799228 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:29.799197 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:31.289940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:31.289904 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:31.662055 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:31.661972 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:09:32.799248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:32.799207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:33.114088 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:33.114013 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:33.593398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:33.593362 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:34.397059 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:34.397026 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:35.806255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:35.806220 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:37.290184 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:37.290148 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:38.659133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:38.659100 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:38.789731 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:38.789697 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:38.890861 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:38.890833 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:40.382354 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:40.382270 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:41.676241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:41.676197 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:09:41.882958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:41.882921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:43.283933 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:43.283899 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:43.679976 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:43.679891 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:44.514862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:44.514827 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44"] Oct 30 04:09:44.554537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:44.553485 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6"] Oct 30 04:09:44.651609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:44.651560 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:44.747974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:44.747929 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:45.003526 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:45.003493 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:45.493056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:45.492970 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:46.592666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:46.592636 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:47.918709 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:47.918669 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:48.494448 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:48.494417 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:49.515683 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:49.515647 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" podUID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" containerName="payload-processing" containerID="cri-o://5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04" gracePeriod=25 Oct 30 04:09:49.554358 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:49.554325 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" podUID="13125b72-bd40-4ec9-b225-54b81ceb78a1" containerName="payload-pre-processing" containerID="cri-o://ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427" gracePeriod=25 Oct 30 04:09:49.987590 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:49.987511 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:51.496175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:51.496140 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:51.595519 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:51.595481 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:51.772947 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:51.772911 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:09:52.326950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:52.326910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:52.698924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:52.698595 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj"] Oct 30 04:09:52.698924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:52.698905 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" containerID="cri-o://e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a" gracePeriod=30 Oct 30 04:09:52.734218 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:52.734186 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:52.941006 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:52.940974 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:53.201346 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:53.201315 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:53.358347 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:53.358300 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" podUID="13125b72-bd40-4ec9-b225-54b81ceb78a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:09:53.358542 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:53.358436 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" podUID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:09:53.550590 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:53.550557 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:53.786478 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:53.786440 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:54.618977 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:54.618947 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:54.848524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:54.848480 4442 patch_prober.go:29] interesting pod/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:09:54.848939 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:54.848539 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:09:54.861299 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:09:54.861268 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:09:55.556550 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:55.556521 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:55.985813 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:55.985781 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:57.466182 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:57.466151 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:57.954278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:57.954249 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:09:58.046978 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.046948 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-token\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047011 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-podinfo\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047048 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-credential-socket\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047080 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-socket\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047126 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-envoy\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047157 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a6560d-6e76-494a-86fd-74216b73ccc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6560d-6e76-494a-86fd-74216b73ccc1-istiod-ca-cert\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047184 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-data\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047229 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-certs\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047654 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047277 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-kube-api-access-bnzrg\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-kube-api-access-bnzrg\") pod \"c5a6560d-6e76-494a-86fd-74216b73ccc1\" (UID: \"c5a6560d-6e76-494a-86fd-74216b73ccc1\") " Oct 30 04:09:58.047654 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047443 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-credential-socket" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:58.047654 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047470 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-socket" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:58.047863 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047669 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.047863 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047689 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.047863 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047825 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a6560d-6e76-494a-86fd-74216b73ccc1-istiod-ca-cert" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:09:58.047863 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047833 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-data" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:58.048016 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.047866 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-certs" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:58.049644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.049611 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-envoy" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:09:58.049901 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.049641 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-token" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:09:58.049901 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.049877 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-podinfo" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:09:58.049999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.049919 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-kube-api-access-bnzrg" pod "c5a6560d-6e76-494a-86fd-74216b73ccc1" (UID: "c5a6560d-6e76-494a-86fd-74216b73ccc1"). InnerVolumeSpecName "kube-api-access-bnzrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:09:58.149043 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.149011 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.149043 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.149039 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.149247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.149053 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5a6560d-6e76-494a-86fd-74216b73ccc1-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.149247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.149062 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.149247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.149070 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5a6560d-6e76-494a-86fd-74216b73ccc1-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.149247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.149078 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnzrg\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-kube-api-access-bnzrg\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.149247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.149086 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5a6560d-6e76-494a-86fd-74216b73ccc1-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:09:58.383817 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.383727 4442 generic.go:356] "Generic (PLEG): container finished" podID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerID="e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a" exitCode=0 Oct 30 04:09:58.383974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.383812 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" Oct 30 04:09:58.383974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.383815 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" event={"ID":"c5a6560d-6e76-494a-86fd-74216b73ccc1","Type":"ContainerDied","Data":"e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a"} Oct 30 04:09:58.383974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.383874 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj" event={"ID":"c5a6560d-6e76-494a-86fd-74216b73ccc1","Type":"ContainerDied","Data":"2a119d08206634c85a4db38221457627dbfafe6245496c0a09da386176b3b438"} Oct 30 04:09:58.383974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.383896 4442 scope.go:122] "RemoveContainer" containerID="e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a" Oct 30 04:09:58.393030 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.393011 4442 scope.go:122] "RemoveContainer" containerID="4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a" Oct 30 04:09:58.400935 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.400919 4442 scope.go:122] "RemoveContainer" containerID="e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a" Oct 30 04:09:58.401196 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:09:58.401176 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a\": container with ID starting with e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a not found: ID does not exist" containerID="e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a" Oct 30 04:09:58.401246 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.401208 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a"} err="failed to get container status \"e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a\": rpc error: code = NotFound desc = could not find container \"e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a\": container with ID starting with e86286c512d81f40331f6fe89e69f352d239d4b7a340c3696a9faa376065604a not found: ID does not exist" Oct 30 04:09:58.401246 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.401224 4442 scope.go:122] "RemoveContainer" containerID="4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a" Oct 30 04:09:58.401449 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:09:58.401432 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a\": container with ID starting with 4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a not found: ID does not exist" containerID="4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a" Oct 30 04:09:58.401492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.401455 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a"} err="failed to get container status \"4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a\": rpc error: code = NotFound desc = could not find container \"4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a\": container with ID starting with 4a7a84aefe46fb7074e1a345c931658d92f05bfaba9552a346dee5da46d5a12a not found: ID does not exist" Oct 30 04:09:58.431540 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.431507 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj"] Oct 30 04:09:58.434951 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.434930 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-30e3dd-openshift-default-fff78f985-8nzfj"] Oct 30 04:09:58.896161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:58.896109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:59.347660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:59.347630 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:59.544717 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:59.544682 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:59.649734 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:09:59.649571 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:00.106672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:00.106628 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" path="/var/lib/kubelet/pods/c5a6560d-6e76-494a-86fd-74216b73ccc1/volumes" Oct 30 04:10:00.490306 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:00.490219 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:01.988457 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:01.988423 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:03.300351 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:03.300316 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:03.358655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:03.358618 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" podUID="13125b72-bd40-4ec9-b225-54b81ceb78a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:03.358655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:03.358640 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" podUID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:03.891957 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:03.891925 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:05.281720 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:05.281688 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:05.814427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:05.814395 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:05.911492 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:05.911459 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:06.503600 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:06.503559 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:06.772554 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:06.772467 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:10:06.798653 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:06.798621 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:06.947864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:06.947824 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z"] Oct 30 04:10:06.948142 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:06.948109 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" containerID="cri-o://ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315" gracePeriod=30 Oct 30 04:10:06.970444 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:06.970397 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:07.904135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:07.904100 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:08.381438 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:08.381402 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:09.793637 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.793605 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:09.796592 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.796568 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:10:09.843807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.843779 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:10:09.864429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.864393 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-kube-api-access-p5wkn\" (UniqueName: \"kubernetes.io/projected/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-kube-api-access-p5wkn\") pod \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " Oct 30 04:10:09.864608 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.864450 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-plugins-config-volume\") pod \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\" (UID: \"8fce5766-abf0-49e6-9f3f-09c2a86f0b81\") " Oct 30 04:10:09.864844 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.864820 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-plugins-config-volume" pod "8fce5766-abf0-49e6-9f3f-09c2a86f0b81" (UID: "8fce5766-abf0-49e6-9f3f-09c2a86f0b81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:09.866876 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.866849 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-kube-api-access-p5wkn" pod "8fce5766-abf0-49e6-9f3f-09c2a86f0b81" (UID: "8fce5766-abf0-49e6-9f3f-09c2a86f0b81"). InnerVolumeSpecName "kube-api-access-p5wkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:09.966357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.966321 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13125b72-bd40-4ec9-b225-54b81ceb78a1-kube-api-access-lzlvl\" (UniqueName: \"kubernetes.io/projected/13125b72-bd40-4ec9-b225-54b81ceb78a1-kube-api-access-lzlvl\") pod \"13125b72-bd40-4ec9-b225-54b81ceb78a1\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " Oct 30 04:10:09.966527 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.966409 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13125b72-bd40-4ec9-b225-54b81ceb78a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13125b72-bd40-4ec9-b225-54b81ceb78a1-plugins-config-volume\") pod \"13125b72-bd40-4ec9-b225-54b81ceb78a1\" (UID: \"13125b72-bd40-4ec9-b225-54b81ceb78a1\") " Oct 30 04:10:09.966620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.966608 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wkn\" (UniqueName: \"kubernetes.io/projected/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-kube-api-access-p5wkn\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:09.966660 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.966623 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fce5766-abf0-49e6-9f3f-09c2a86f0b81-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:09.966845 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.966821 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13125b72-bd40-4ec9-b225-54b81ceb78a1-plugins-config-volume" pod "13125b72-bd40-4ec9-b225-54b81ceb78a1" (UID: "13125b72-bd40-4ec9-b225-54b81ceb78a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:09.968835 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:09.968812 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13125b72-bd40-4ec9-b225-54b81ceb78a1-kube-api-access-lzlvl" pod "13125b72-bd40-4ec9-b225-54b81ceb78a1" (UID: "13125b72-bd40-4ec9-b225-54b81ceb78a1"). InnerVolumeSpecName "kube-api-access-lzlvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:10.067260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.067179 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzlvl\" (UniqueName: \"kubernetes.io/projected/13125b72-bd40-4ec9-b225-54b81ceb78a1-kube-api-access-lzlvl\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.067260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.067207 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13125b72-bd40-4ec9-b225-54b81ceb78a1-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:10.440178 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.440088 4442 generic.go:356] "Generic (PLEG): container finished" podID="13125b72-bd40-4ec9-b225-54b81ceb78a1" containerID="ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427" exitCode=0 Oct 30 04:10:10.440178 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.440134 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" Oct 30 04:10:10.440178 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.440170 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" event={"ID":"13125b72-bd40-4ec9-b225-54b81ceb78a1","Type":"ContainerDied","Data":"ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427"} Oct 30 04:10:10.440454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.440224 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6" event={"ID":"13125b72-bd40-4ec9-b225-54b81ceb78a1","Type":"ContainerDied","Data":"b8d6b87bc80ed84f0982ba747c60d63dc81bd7b2f4aab3568bdc358bdb4c6da8"} Oct 30 04:10:10.440454 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.440245 4442 scope.go:122] "RemoveContainer" containerID="ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427" Oct 30 04:10:10.441796 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.441739 4442 generic.go:356] "Generic (PLEG): container finished" podID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" containerID="5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04" exitCode=0 Oct 30 04:10:10.442096 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.441792 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" event={"ID":"8fce5766-abf0-49e6-9f3f-09c2a86f0b81","Type":"ContainerDied","Data":"5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04"} Oct 30 04:10:10.442096 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.441824 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" Oct 30 04:10:10.442096 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.441873 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44" event={"ID":"8fce5766-abf0-49e6-9f3f-09c2a86f0b81","Type":"ContainerDied","Data":"b2a6e3e5d5179e2a25597e0c77967db3b3e7a76b6d8591b97353fc3245423391"} Oct 30 04:10:10.449219 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.449199 4442 scope.go:122] "RemoveContainer" containerID="ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427" Oct 30 04:10:10.449490 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:10:10.449469 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427\": container with ID starting with ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427 not found: ID does not exist" containerID="ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427" Oct 30 04:10:10.449593 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.449495 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427"} err="failed to get container status \"ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427\": rpc error: code = NotFound desc = could not find container \"ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427\": container with ID starting with ce353456bb1fb185c24527321943e42583e893108d9d4ecda1747d3ea4299427 not found: ID does not exist" Oct 30 04:10:10.449593 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.449509 4442 scope.go:122] "RemoveContainer" containerID="5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04" Oct 30 04:10:10.457579 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.457563 4442 scope.go:122] "RemoveContainer" containerID="5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04" Oct 30 04:10:10.457832 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:10:10.457815 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04\": container with ID starting with 5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04 not found: ID does not exist" containerID="5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04" Oct 30 04:10:10.457897 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.457838 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04"} err="failed to get container status \"5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04\": rpc error: code = NotFound desc = could not find container \"5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04\": container with ID starting with 5d67208509e0d91ba6a23d4d01485e86e2669f06ed5d17a3317a79cfaf322c04 not found: ID does not exist" Oct 30 04:10:10.485672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.485647 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6"] Oct 30 04:10:10.496443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.496413 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-30e3dd-7556fbf79-vl7g6"] Oct 30 04:10:10.527229 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.527192 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44"] Oct 30 04:10:10.534332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:10.534310 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-30e3dd-58f9bf77fd-bcn44"] Oct 30 04:10:11.390999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:11.390963 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:12.108140 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.108098 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13125b72-bd40-4ec9-b225-54b81ceb78a1" path="/var/lib/kubelet/pods/13125b72-bd40-4ec9-b225-54b81ceb78a1/volumes" Oct 30 04:10:12.108666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.108642 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" path="/var/lib/kubelet/pods/8fce5766-abf0-49e6-9f3f-09c2a86f0b81/volumes" Oct 30 04:10:12.195437 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.195413 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:10:12.291496 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291462 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-socket\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.291692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291509 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd461436-cb68-4758-8592-62744af1dd53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd461436-cb68-4758-8592-62744af1dd53-istiod-ca-cert\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.291692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291544 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-istio-token\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-istio-token\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.291692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291581 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-data\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.291692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291597 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-envoy\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.291692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291662 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd461436-cb68-4758-8592-62744af1dd53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd461436-cb68-4758-8592-62744af1dd53-istio-podinfo\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.292042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291728 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-certs\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.292042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291783 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-credential-socket\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.292042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291786 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-socket" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:12.292042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291820 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-kube-api-access-xtrcq\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-kube-api-access-xtrcq\") pod \"bd461436-cb68-4758-8592-62744af1dd53\" (UID: \"bd461436-cb68-4758-8592-62744af1dd53\") " Oct 30 04:10:12.292042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.291912 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-data" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:12.292042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.292023 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd461436-cb68-4758-8592-62744af1dd53-istiod-ca-cert" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:12.292383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.292115 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-certs" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:12.292383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.292274 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.292383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.292296 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.292383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.292306 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-credential-socket" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:12.292383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.292311 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd461436-cb68-4758-8592-62744af1dd53-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.292383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.292347 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.294284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.294259 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd461436-cb68-4758-8592-62744af1dd53-istio-podinfo" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:10:12.294411 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.294358 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-kube-api-access-xtrcq" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "kube-api-access-xtrcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:12.294502 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.294441 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-envoy" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:10:12.294502 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.294482 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-istio-token" pod "bd461436-cb68-4758-8592-62744af1dd53" (UID: "bd461436-cb68-4758-8592-62744af1dd53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:12.393772 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.393720 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.393772 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.393745 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.393772 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.393773 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd461436-cb68-4758-8592-62744af1dd53-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.394208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.393782 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd461436-cb68-4758-8592-62744af1dd53-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.394208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.393791 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtrcq\" (UniqueName: \"kubernetes.io/projected/bd461436-cb68-4758-8592-62744af1dd53-kube-api-access-xtrcq\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:12.455744 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.455704 4442 generic.go:356] "Generic (PLEG): container finished" podID="bd461436-cb68-4758-8592-62744af1dd53" containerID="ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315" exitCode=0 Oct 30 04:10:12.455921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.455772 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" Oct 30 04:10:12.455921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.455800 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" event={"ID":"bd461436-cb68-4758-8592-62744af1dd53","Type":"ContainerDied","Data":"ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315"} Oct 30 04:10:12.455921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.455862 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z" event={"ID":"bd461436-cb68-4758-8592-62744af1dd53","Type":"ContainerDied","Data":"320cd1d588f426ce55057968e24155de5fd156d1f84b5169721f9bb6a798d4d4"} Oct 30 04:10:12.455921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.455885 4442 scope.go:122] "RemoveContainer" containerID="ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315" Oct 30 04:10:12.465501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.465480 4442 scope.go:122] "RemoveContainer" containerID="ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315" Oct 30 04:10:12.465786 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:10:12.465763 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315\": container with ID starting with ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315 not found: ID does not exist" containerID="ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315" Oct 30 04:10:12.465841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.465798 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315"} err="failed to get container status \"ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315\": rpc error: code = NotFound desc = could not find container \"ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315\": container with ID starting with ca9d6a133480ab8bd62eb884ff4ef0d3605bf4881d471bfee542a97323a82315 not found: ID does not exist" Oct 30 04:10:12.501368 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.501343 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z"] Oct 30 04:10:12.507094 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.507070 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-dd81fd-openshift-default-74ddf69555-72m8z"] Oct 30 04:10:12.796936 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:12.796906 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:13.269685 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:13.269655 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s"] Oct 30 04:10:13.310476 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:13.310449 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:13.394994 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:13.394966 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:13.497023 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:13.496983 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:14.107335 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:14.107303 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd461436-cb68-4758-8592-62744af1dd53" path="/var/lib/kubelet/pods/bd461436-cb68-4758-8592-62744af1dd53/volumes" Oct 30 04:10:14.440376 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:14.440290 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:14.991388 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:14.991355 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:15.991976 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:15.991943 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:17.184543 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:17.184516 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:17.734322 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:17.734205 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:18.270281 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:18.270237 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" podUID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" containerName="payload-processing" containerID="cri-o://9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7" gracePeriod=25 Oct 30 04:10:19.286814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:19.286774 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:20.429435 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:20.429397 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:10:20.706040 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:20.705956 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:21.033913 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:21.033879 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:21.133968 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:21.133919 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:21.718718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:21.718669 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:22.098249 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:22.098196 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:22.361714 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:22.361639 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:22.933329 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:22.933288 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:23.893479 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:23.893449 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:24.851648 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:24.851610 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" podUID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:24.860898 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:10:24.860864 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:10:25.394999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:25.394965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:26.801672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:26.801642 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:28.400564 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:28.400521 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:28.601417 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:28.601378 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2"] Oct 30 04:10:28.825523 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:28.825491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:28.926495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:28.926464 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:29.604472 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:29.604433 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:29.985226 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:29.985144 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:31.270222 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:31.270188 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:31.804279 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:31.804246 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:33.280202 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:33.280165 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:33.602701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:33.602608 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" podUID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" containerName="payload-processing" containerID="cri-o://1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a" gracePeriod=25 Oct 30 04:10:34.333481 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:34.333447 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:10:34.852127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:34.852092 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" podUID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:34.878514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:34.878480 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:36.237356 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:36.237323 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:36.337875 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:36.337830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:36.434630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:36.434595 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" podUID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:36.489325 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:36.489248 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:36.915709 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:36.915673 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:37.315173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:37.315133 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:37.854769 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:37.854726 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:38.092086 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.092049 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:38.526334 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.526277 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 04:10:38.558089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.558066 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-kube-api-access-j7stm\" (UniqueName: \"kubernetes.io/projected/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-kube-api-access-j7stm\") pod \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " Oct 30 04:10:38.558211 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.558109 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-plugins-config-volume\") pod \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\" (UID: \"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29\") " Oct 30 04:10:38.558475 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.558456 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-plugins-config-volume" pod "1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" (UID: "1480f37f-0d88-4fcf-8a8f-0a18a11cfc29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:38.560208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.560187 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-kube-api-access-j7stm" pod "1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" (UID: "1480f37f-0d88-4fcf-8a8f-0a18a11cfc29"). InnerVolumeSpecName "kube-api-access-j7stm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:38.582489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.582464 4442 generic.go:356] "Generic (PLEG): container finished" podID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" containerID="9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7" exitCode=0 Oct 30 04:10:38.582615 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.582505 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" Oct 30 04:10:38.582615 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.582550 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" event={"ID":"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29","Type":"ContainerDied","Data":"9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7"} Oct 30 04:10:38.582615 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.582602 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s" event={"ID":"1480f37f-0d88-4fcf-8a8f-0a18a11cfc29","Type":"ContainerDied","Data":"3feb8feb07ae8ec3c09c2f9404def923af3c173c4a2920962cb234c7a2f0258d"} Oct 30 04:10:38.582739 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.582630 4442 scope.go:122] "RemoveContainer" containerID="9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7" Oct 30 04:10:38.591899 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.591878 4442 scope.go:122] "RemoveContainer" containerID="9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7" Oct 30 04:10:38.592226 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:10:38.592205 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7\": container with ID starting with 9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7 not found: ID does not exist" containerID="9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7" Oct 30 04:10:38.592288 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.592236 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7"} err="failed to get container status \"9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7\": rpc error: code = NotFound desc = could not find container \"9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7\": container with ID starting with 9aed480e799af8ef9f43fd984c330855641c1d8e69a309efe94c1b50e03e02b7 not found: ID does not exist" Oct 30 04:10:38.629256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.629231 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s"] Oct 30 04:10:38.635650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.635626 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-398c25-654bf4cb5b-tm94s"] Oct 30 04:10:38.659839 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.659811 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7stm\" (UniqueName: \"kubernetes.io/projected/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-kube-api-access-j7stm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:38.659839 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:38.659835 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:39.685422 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:39.685391 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:40.105743 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:40.105712 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" path="/var/lib/kubelet/pods/1480f37f-0d88-4fcf-8a8f-0a18a11cfc29/volumes" Oct 30 04:10:41.186929 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:41.186894 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:42.680872 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:42.680841 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:42.922732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:42.922701 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:43.523281 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:43.523244 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:44.103148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:44.103107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:45.050041 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:45.050009 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:10:45.615180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:45.615130 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:46.434991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:46.434952 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" podUID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:10:47.124466 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:47.124415 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:48.481980 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:48.481945 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:49.107638 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:49.107604 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:49.190486 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:49.190452 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:49.901524 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:49.901476 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:50.090177 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:50.090145 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:52.911436 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:52.911405 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:53.053000 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:53.052955 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:53.850728 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:53.850703 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 04:10:53.911930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:53.911857 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:54.008422 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.008391 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-kube-api-access-x5pm4\" (UniqueName: \"kubernetes.io/projected/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-kube-api-access-x5pm4\") pod \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " Oct 30 04:10:54.008593 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.008454 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-plugins-config-volume\") pod \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\" (UID: \"fe013b5e-3ce2-41f1-860d-7ce69690c0a8\") " Oct 30 04:10:54.008881 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.008859 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-plugins-config-volume" pod "fe013b5e-3ce2-41f1-860d-7ce69690c0a8" (UID: "fe013b5e-3ce2-41f1-860d-7ce69690c0a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:10:54.010782 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.010748 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-kube-api-access-x5pm4" pod "fe013b5e-3ce2-41f1-860d-7ce69690c0a8" (UID: "fe013b5e-3ce2-41f1-860d-7ce69690c0a8"). InnerVolumeSpecName "kube-api-access-x5pm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:10:54.109260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.109235 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pm4\" (UniqueName: \"kubernetes.io/projected/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-kube-api-access-x5pm4\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:54.109260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.109258 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe013b5e-3ce2-41f1-860d-7ce69690c0a8-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:10:54.668970 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.668937 4442 generic.go:356] "Generic (PLEG): container finished" podID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" containerID="1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a" exitCode=0 Oct 30 04:10:54.669175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.669000 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" Oct 30 04:10:54.669175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.669016 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" event={"ID":"fe013b5e-3ce2-41f1-860d-7ce69690c0a8","Type":"ContainerDied","Data":"1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a"} Oct 30 04:10:54.669175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.669058 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2" event={"ID":"fe013b5e-3ce2-41f1-860d-7ce69690c0a8","Type":"ContainerDied","Data":"766996063c8984685fbe0e4d32aa91528d9c5dcf9808c82462c72143eaf76da5"} Oct 30 04:10:54.669175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.669078 4442 scope.go:122] "RemoveContainer" containerID="1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a" Oct 30 04:10:54.677918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.677895 4442 scope.go:122] "RemoveContainer" containerID="1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a" Oct 30 04:10:54.678200 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:10:54.678175 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a\": container with ID starting with 1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a not found: ID does not exist" containerID="1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a" Oct 30 04:10:54.678240 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.678216 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a"} err="failed to get container status \"1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a\": rpc error: code = NotFound desc = could not find container \"1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a\": container with ID starting with 1d378ac504350fc78088c4d545dc1f2bddccef1927a7a79c8530698b8c8e8f6a not found: ID does not exist" Oct 30 04:10:54.715120 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.715094 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2"] Oct 30 04:10:54.726309 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:54.726280 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-855e14-56ff45ddcd-gjvz2"] Oct 30 04:10:54.859720 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:10:54.859684 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:10:55.652314 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:55.652276 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:55.753497 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:55.753461 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:56.112410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:56.112370 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" path="/var/lib/kubelet/pods/fe013b5e-3ce2-41f1-860d-7ce69690c0a8/volumes" Oct 30 04:10:59.480477 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:59.480419 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:59.580870 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:10:59.580840 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:01.198161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.198129 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:11:01.288044 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.288011 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh"] Oct 30 04:11:01.288244 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.288225 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" podUID="7f2295a7-6365-4457-8278-7bd0c1f3511c" containerName="maas-api" containerID="cri-o://a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543" gracePeriod=30 Oct 30 04:11:01.342137 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.342109 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:01.549443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.549416 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 04:11:01.673801 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.673744 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-metrics-certs\") pod \"7f2295a7-6365-4457-8278-7bd0c1f3511c\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " Oct 30 04:11:01.673997 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.673825 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2295a7-6365-4457-8278-7bd0c1f3511c-kube-api-access-p9zvx\" (UniqueName: \"kubernetes.io/projected/7f2295a7-6365-4457-8278-7bd0c1f3511c-kube-api-access-p9zvx\") pod \"7f2295a7-6365-4457-8278-7bd0c1f3511c\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " Oct 30 04:11:01.673997 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.673846 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2295a7-6365-4457-8278-7bd0c1f3511c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2295a7-6365-4457-8278-7bd0c1f3511c-openshift-service-ca\") pod \"7f2295a7-6365-4457-8278-7bd0c1f3511c\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " Oct 30 04:11:01.673997 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.673906 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-maas-api-tls\") pod \"7f2295a7-6365-4457-8278-7bd0c1f3511c\" (UID: \"7f2295a7-6365-4457-8278-7bd0c1f3511c\") " Oct 30 04:11:01.674268 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.674238 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2295a7-6365-4457-8278-7bd0c1f3511c-openshift-service-ca" pod "7f2295a7-6365-4457-8278-7bd0c1f3511c" (UID: "7f2295a7-6365-4457-8278-7bd0c1f3511c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:01.676225 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.676196 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-maas-api-tls" pod "7f2295a7-6365-4457-8278-7bd0c1f3511c" (UID: "7f2295a7-6365-4457-8278-7bd0c1f3511c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:01.676332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.676221 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-metrics-certs" pod "7f2295a7-6365-4457-8278-7bd0c1f3511c" (UID: "7f2295a7-6365-4457-8278-7bd0c1f3511c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:01.676436 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.676418 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2295a7-6365-4457-8278-7bd0c1f3511c-kube-api-access-p9zvx" pod "7f2295a7-6365-4457-8278-7bd0c1f3511c" (UID: "7f2295a7-6365-4457-8278-7bd0c1f3511c"). InnerVolumeSpecName "kube-api-access-p9zvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:01.703992 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.703959 4442 generic.go:356] "Generic (PLEG): container finished" podID="7f2295a7-6365-4457-8278-7bd0c1f3511c" containerID="a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543" exitCode=0 Oct 30 04:11:01.704196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.704009 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" Oct 30 04:11:01.704196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.704016 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" event={"ID":"7f2295a7-6365-4457-8278-7bd0c1f3511c","Type":"ContainerDied","Data":"a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543"} Oct 30 04:11:01.704196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.704064 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh" event={"ID":"7f2295a7-6365-4457-8278-7bd0c1f3511c","Type":"ContainerDied","Data":"ab5a9f4b7db990f48792478eb10c2ddb1f7b15665f2942b0a51298ec7a1d46cf"} Oct 30 04:11:01.704196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.704090 4442 scope.go:122] "RemoveContainer" containerID="a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543" Oct 30 04:11:01.714835 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.714812 4442 scope.go:122] "RemoveContainer" containerID="a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543" Oct 30 04:11:01.715171 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:01.715146 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543\": container with ID starting with a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543 not found: ID does not exist" containerID="a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543" Oct 30 04:11:01.715254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.715186 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543"} err="failed to get container status \"a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543\": rpc error: code = NotFound desc = could not find container \"a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543\": container with ID starting with a508539da2b6bfce2f9d620ab8dcf297b149669f66c1e00665ace7b563cc7543 not found: ID does not exist" Oct 30 04:11:01.751317 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.751237 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh"] Oct 30 04:11:01.752570 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.752549 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2d5b09-596b49549b-7jgxh"] Oct 30 04:11:01.775326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.775300 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:01.775326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.775324 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zvx\" (UniqueName: \"kubernetes.io/projected/7f2295a7-6365-4457-8278-7bd0c1f3511c-kube-api-access-p9zvx\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:01.775488 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.775334 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2295a7-6365-4457-8278-7bd0c1f3511c-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:01.775488 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.775344 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2295a7-6365-4457-8278-7bd0c1f3511c-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:01.960228 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:01.959959 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:02.059330 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:02.059294 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:02.106159 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:02.106119 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2295a7-6365-4457-8278-7bd0c1f3511c" path="/var/lib/kubelet/pods/7f2295a7-6365-4457-8278-7bd0c1f3511c/volumes" Oct 30 04:11:02.634532 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:02.634498 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:03.063590 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:03.063530 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm"] Oct 30 04:11:03.064040 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:03.063987 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" containerID="cri-o://f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357" gracePeriod=30 Oct 30 04:11:03.102201 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:03.102145 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:05.500499 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:05.500468 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ndw77_874a2d48-0e3f-45c1-bce3-bfccb9d006b8/limitador/0.log" Oct 30 04:11:06.725184 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:06.725154 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:11:08.329521 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.329498 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 04:11:08.438107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438019 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-envoy\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438066 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-certs\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438100 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f428cce2-c5cc-4903-bbeb-898b1a84d249-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f428cce2-c5cc-4903-bbeb-898b1a84d249-istiod-ca-cert\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438118 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-credential-socket\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438150 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-token\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-token\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438180 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-data\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438427 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438381 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-socket\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438456 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-certs" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:08.438662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438493 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-credential-socket" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:08.438662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438431 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-kube-api-access-694d7\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-kube-api-access-694d7\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.438662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438579 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-socket" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:08.438662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438603 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f428cce2-c5cc-4903-bbeb-898b1a84d249-istiod-ca-cert" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:08.438662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438606 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-podinfo\") pod \"f428cce2-c5cc-4903-bbeb-898b1a84d249\" (UID: \"f428cce2-c5cc-4903-bbeb-898b1a84d249\") " Oct 30 04:11:08.439009 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.438696 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-data" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:08.439139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.439118 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.439205 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.439144 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f428cce2-c5cc-4903-bbeb-898b1a84d249-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.439205 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.439157 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.439205 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.439170 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.439205 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.439184 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.441088 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.441064 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-envoy" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:08.441223 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.441100 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-token" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:08.441407 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.441376 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-podinfo" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:11:08.441510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.441433 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-kube-api-access-694d7" pod "f428cce2-c5cc-4903-bbeb-898b1a84d249" (UID: "f428cce2-c5cc-4903-bbeb-898b1a84d249"). InnerVolumeSpecName "kube-api-access-694d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:08.539937 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.539911 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.540089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.539942 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.540089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.539958 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-694d7\" (UniqueName: \"kubernetes.io/projected/f428cce2-c5cc-4903-bbeb-898b1a84d249-kube-api-access-694d7\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.540089 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.539973 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f428cce2-c5cc-4903-bbeb-898b1a84d249-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:08.734643 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.734564 4442 generic.go:356] "Generic (PLEG): container finished" podID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerID="f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357" exitCode=0 Oct 30 04:11:08.734643 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.734617 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" Oct 30 04:11:08.734643 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.734635 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" event={"ID":"f428cce2-c5cc-4903-bbeb-898b1a84d249","Type":"ContainerDied","Data":"f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357"} Oct 30 04:11:08.734891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.734683 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm" event={"ID":"f428cce2-c5cc-4903-bbeb-898b1a84d249","Type":"ContainerDied","Data":"6b64f6e62a7c793dd86b3094937db8d961bf4c0c3e604babfe7184886b229668"} Oct 30 04:11:08.734891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.734707 4442 scope.go:122] "RemoveContainer" containerID="f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357" Oct 30 04:11:08.745681 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.745659 4442 scope.go:122] "RemoveContainer" containerID="e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb" Oct 30 04:11:08.754453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.754435 4442 scope.go:122] "RemoveContainer" containerID="f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357" Oct 30 04:11:08.754742 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:08.754713 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357\": container with ID starting with f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357 not found: ID does not exist" containerID="f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357" Oct 30 04:11:08.754853 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.754779 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357"} err="failed to get container status \"f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357\": rpc error: code = NotFound desc = could not find container \"f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357\": container with ID starting with f630e4c0ec2ac29ca93415a4be9c0e01401d1ffaacd90c9440c6c42f45389357 not found: ID does not exist" Oct 30 04:11:08.754853 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.754809 4442 scope.go:122] "RemoveContainer" containerID="e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb" Oct 30 04:11:08.755148 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:08.755128 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb\": container with ID starting with e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb not found: ID does not exist" containerID="e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb" Oct 30 04:11:08.755203 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.755153 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb"} err="failed to get container status \"e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb\": rpc error: code = NotFound desc = could not find container \"e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb\": container with ID starting with e4f0fdb506536e9ef22a33dcd9915529dcb0e4b51058d9b7f7e15b8520fa0fcb not found: ID does not exist" Oct 30 04:11:08.781843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.781822 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm"] Oct 30 04:11:08.785257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.785234 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2d5b09-openshift-default-6f4c95fc49-ssczm"] Oct 30 04:11:08.910842 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:08.910816 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-4dk5t_05381e85-4985-4f92-9d08-6cca60bf8059/manager/1.log" Oct 30 04:11:09.838411 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:09.838329 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:11:10.106339 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:10.106264 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" path="/var/lib/kubelet/pods/f428cce2-c5cc-4903-bbeb-898b1a84d249/volumes" Oct 30 04:11:10.602277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:10.602251 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-4dk5t_05381e85-4985-4f92-9d08-6cca60bf8059/manager/0.log" Oct 30 04:11:18.045044 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.045008 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:18.534107 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534073 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj"] Oct 30 04:11:18.534440 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534426 4442 cpu_manager.go:414] "containerMap: removing container" podUID="193ae444-3be2-447a-a65e-90d467725718" containerName="payload-pre-processing" Oct 30 04:11:18.534509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534441 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193ae444-3be2-447a-a65e-90d467725718" containerName="payload-pre-processing" Oct 30 04:11:18.534509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534456 4442 cpu_manager.go:414] "containerMap: removing container" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" Oct 30 04:11:18.534509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534467 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" Oct 30 04:11:18.534509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534478 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" containerName="maas-api" Oct 30 04:11:18.534509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534485 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" containerName="maas-api" Oct 30 04:11:18.534509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534498 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" Oct 30 04:11:18.534509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534505 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534516 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534523 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534537 4442 cpu_manager.go:414] "containerMap: removing container" podUID="14b80988-fbb8-4d18-910b-190fc3c6bcdd" containerName="maas-api" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534544 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b80988-fbb8-4d18-910b-190fc3c6bcdd" containerName="maas-api" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534554 4442 cpu_manager.go:414] "containerMap: removing container" podUID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" containerName="payload-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534562 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" containerName="payload-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534574 4442 cpu_manager.go:414] "containerMap: removing container" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534579 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534588 4442 cpu_manager.go:414] "containerMap: removing container" podUID="845e7f17-4711-4381-9302-7a3568a6e48d" containerName="payload-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534595 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845e7f17-4711-4381-9302-7a3568a6e48d" containerName="payload-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534600 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534606 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534621 4442 cpu_manager.go:414] "containerMap: removing container" podUID="13125b72-bd40-4ec9-b225-54b81ceb78a1" containerName="payload-pre-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534629 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13125b72-bd40-4ec9-b225-54b81ceb78a1" containerName="payload-pre-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534657 4442 cpu_manager.go:414] "containerMap: removing container" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534662 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534670 4442 cpu_manager.go:414] "containerMap: removing container" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534675 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534683 4442 cpu_manager.go:414] "containerMap: removing container" podUID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" containerName="payload-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534687 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" containerName="payload-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534694 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7f2295a7-6365-4457-8278-7bd0c1f3511c" containerName="maas-api" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534698 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2295a7-6365-4457-8278-7bd0c1f3511c" containerName="maas-api" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534703 4442 cpu_manager.go:414] "containerMap: removing container" podUID="5c36d571-4774-49b7-8e1e-25821eb776d5" containerName="maas-api" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534707 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c36d571-4774-49b7-8e1e-25821eb776d5" containerName="maas-api" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534723 4442 cpu_manager.go:414] "containerMap: removing container" podUID="d5452058-93bf-4de6-a642-2dedbad09bd2" containerName="main" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534728 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5452058-93bf-4de6-a642-2dedbad09bd2" containerName="main" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534735 4442 cpu_manager.go:414] "containerMap: removing container" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534739 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534746 4442 cpu_manager.go:414] "containerMap: removing container" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534767 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb022099-cfd2-41b8-aabc-c6cb77f40112" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534783 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534789 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534794 4442 cpu_manager.go:414] "containerMap: removing container" podUID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" containerName="payload-processing" Oct 30 04:11:18.534790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534799 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" containerName="payload-processing" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534878 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534886 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd461436-cb68-4758-8592-62744af1dd53" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534891 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe013b5e-3ce2-41f1-860d-7ce69690c0a8" containerName="payload-processing" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534898 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2295a7-6365-4457-8278-7bd0c1f3511c" containerName="maas-api" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534903 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c36d571-4774-49b7-8e1e-25821eb776d5" containerName="maas-api" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534910 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5452058-93bf-4de6-a642-2dedbad09bd2" containerName="main" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534915 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534921 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534930 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7e4e5c-5372-4cea-aa2f-8d63671314de" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534937 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fce5766-abf0-49e6-9f3f-09c2a86f0b81" containerName="payload-processing" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534942 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="193ae444-3be2-447a-a65e-90d467725718" containerName="payload-pre-processing" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534948 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534954 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b86d4db2-5c40-4ff1-a390-6dbf6c8204e1" containerName="maas-api" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534959 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f428cce2-c5cc-4903-bbeb-898b1a84d249" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534964 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ebcfb-6be1-47de-95a2-4b8876205c34" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534970 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b80988-fbb8-4d18-910b-190fc3c6bcdd" containerName="maas-api" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534976 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="1480f37f-0d88-4fcf-8a8f-0a18a11cfc29" containerName="payload-processing" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534981 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9cbdcf-886d-46e5-a7cd-7321e55e1a68" containerName="istio-proxy" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534986 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="845e7f17-4711-4381-9302-7a3568a6e48d" containerName="payload-processing" Oct 30 04:11:18.536148 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.534992 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="13125b72-bd40-4ec9-b225-54b81ceb78a1" containerName="payload-pre-processing" Oct 30 04:11:18.542778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.540707 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.556405 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.556378 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6e1eb434-openshift-default-dockercfg-9rvdl\"" Oct 30 04:11:18.569030 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.569005 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj"] Oct 30 04:11:18.584737 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.584702 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:18.622668 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622641 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdwj\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-kube-api-access-rvdwj\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.622801 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622681 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4554f90-1135-42c0-bb31-917f17d3c458-istiod-ca-cert\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.622801 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622746 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-certs\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.622891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622854 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-data\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.622891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622882 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-envoy\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.622962 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622902 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-credential-socket\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.622962 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622929 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-socket\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.622962 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.622949 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-istio-token\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.623063 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.623029 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4554f90-1135-42c0-bb31-917f17d3c458-istio-podinfo\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724123 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724083 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-credential-socket\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724123 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724130 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-socket\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724151 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-istio-token\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724187 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4554f90-1135-42c0-bb31-917f17d3c458-istio-podinfo\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724235 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdwj\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-kube-api-access-rvdwj\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724270 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4554f90-1135-42c0-bb31-917f17d3c458-istiod-ca-cert\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724313 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-certs\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724362 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-data\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724393 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-envoy\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724569 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-credential-socket\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724645 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-socket\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.724996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724971 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-certs\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.725076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.724984 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-data\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.725199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.725178 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4554f90-1135-42c0-bb31-917f17d3c458-istiod-ca-cert\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.726987 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.726961 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-envoy\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.727094 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.727079 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4554f90-1135-42c0-bb31-917f17d3c458-istio-podinfo\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.741913 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.741884 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdwj\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-kube-api-access-rvdwj\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.741913 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.741890 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-istio-token\") pod \"e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:18.781597 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.781567 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:18.856100 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:18.856004 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:19.167964 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.167943 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj"] Oct 30 04:11:19.169888 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:11:19.169854 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4554f90_1135_42c0_bb31_917f17d3c458.slice/crio-878e9972240228e382e51a7b73a6ceb79b2647c4cf3139d8ef21a09ae1aabc5b WatchSource:0}: Error finding container 878e9972240228e382e51a7b73a6ceb79b2647c4cf3139d8ef21a09ae1aabc5b: Status 404 returned error can't find the container with id 878e9972240228e382e51a7b73a6ceb79b2647c4cf3139d8ef21a09ae1aabc5b Oct 30 04:11:19.172051 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.172019 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:11:19.172144 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.172080 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:11:19.172144 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.172109 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:11:19.786248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.786213 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" event={"ID":"b4554f90-1135-42c0-bb31-917f17d3c458","Type":"ContainerStarted","Data":"9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2"} Oct 30 04:11:19.786429 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.786271 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" event={"ID":"b4554f90-1135-42c0-bb31-917f17d3c458","Type":"ContainerStarted","Data":"878e9972240228e382e51a7b73a6ceb79b2647c4cf3139d8ef21a09ae1aabc5b"} Oct 30 04:11:19.837241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.837182 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podStartSLOduration=1.837163316 podStartE2EDuration="1.837163316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:19.836113726 +0000 UTC m=+4103.838466924" watchObservedRunningTime="2026-10-30 04:11:19.837163316 +0000 UTC m=+4103.839516516" Oct 30 04:11:19.840463 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.840439 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:19.856164 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:19.856139 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:20.018157 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:20.018124 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:20.124196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:20.124114 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:20.862472 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:20.862442 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:21.204212 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:21.204127 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:21.652266 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:21.652226 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:21.750628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:21.750595 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:21.795512 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:21.795473 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:21.796671 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:21.796651 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:23.005290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.005255 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6"] Oct 30 04:11:23.005795 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.005619 4442 cpu_manager.go:414] "containerMap: removing container" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" Oct 30 04:11:23.005795 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.005630 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107b2e13-5fb2-4241-8143-e75258d44261" containerName="istio-proxy" Oct 30 04:11:23.005795 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.005685 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a6560d-6e76-494a-86fd-74216b73ccc1" containerName="istio-proxy" Oct 30 04:11:23.013199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.013173 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.038186 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.038156 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6e1eb434-serving-cert\"" Oct 30 04:11:23.039954 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.039929 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6"] Oct 30 04:11:23.063943 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.063905 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-metrics-certs\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.064106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.063984 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79z8f\" (UniqueName: \"kubernetes.io/projected/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-kube-api-access-79z8f\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.064106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.064025 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-openshift-service-ca\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.064239 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.064119 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.075783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.075737 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:23.165538 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.165505 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.165675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.165568 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-metrics-certs\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.165675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.165630 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79z8f\" (UniqueName: \"kubernetes.io/projected/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-kube-api-access-79z8f\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.165675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.165647 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-openshift-service-ca\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.165675 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:23.165660 4442 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-serving-cert: secret "maas-api-e2e-ait-6e1eb434-serving-cert" not found Oct 30 04:11:23.165952 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:23.165742 4442 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls podName:7c94d9d5-97ac-4930-9f88-7e83fbc37ef6 nodeName:}" failed. No retries permitted until 2026-10-30 04:11:23.665721599 +0000 UTC m=+4107.668074790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls") pod "maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" (UID: "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6") : secret "maas-api-e2e-ait-6e1eb434-serving-cert" not found Oct 30 04:11:23.166495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.166461 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-openshift-service-ca\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.168290 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.168269 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-metrics-certs\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.176106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.176060 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:23.186835 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.186781 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79z8f\" (UniqueName: \"kubernetes.io/projected/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-kube-api-access-79z8f\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.325108 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.325024 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:23.425740 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.425706 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:23.481073 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.481037 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz"] Oct 30 04:11:23.484582 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.484564 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.500325 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.500303 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6e1eb434\"" Oct 30 04:11:23.503278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.503256 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6e1eb434-dockercfg-m457h\"" Oct 30 04:11:23.504917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.504896 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz"] Oct 30 04:11:23.521609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.521582 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:23.543960 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.543936 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7"] Oct 30 04:11:23.547386 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.547370 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.565387 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.565364 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7"] Oct 30 04:11:23.570145 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.570123 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t92z5\" (UniqueName: \"kubernetes.io/projected/a581c127-5a40-423e-8485-032317cce6ad-kube-api-access-t92z5\") pod \"payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.570252 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.570163 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a581c127-5a40-423e-8485-032317cce6ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.570252 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.570191 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrs2t\" (UniqueName: \"kubernetes.io/projected/b11b14d1-1342-447a-9494-d3b6d4a40de1-kube-api-access-qrs2t\") pod \"payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.570437 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.570417 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11b14d1-1342-447a-9494-d3b6d4a40de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.671736 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.671665 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11b14d1-1342-447a-9494-d3b6d4a40de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.671736 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.671717 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.672130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.671740 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t92z5\" (UniqueName: \"kubernetes.io/projected/a581c127-5a40-423e-8485-032317cce6ad-kube-api-access-t92z5\") pod \"payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.672130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.671779 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a581c127-5a40-423e-8485-032317cce6ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.672130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.671803 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrs2t\" (UniqueName: \"kubernetes.io/projected/b11b14d1-1342-447a-9494-d3b6d4a40de1-kube-api-access-qrs2t\") pod \"payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.672613 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.672589 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11b14d1-1342-447a-9494-d3b6d4a40de1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.672658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.672589 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a581c127-5a40-423e-8485-032317cce6ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.674284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.674265 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls\") pod \"maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:23.692206 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.692179 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t92z5\" (UniqueName: \"kubernetes.io/projected/a581c127-5a40-423e-8485-032317cce6ad-kube-api-access-t92z5\") pod \"payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.692410 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.692391 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrs2t\" (UniqueName: \"kubernetes.io/projected/b11b14d1-1342-447a-9494-d3b6d4a40de1-kube-api-access-qrs2t\") pod \"payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.799740 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.799494 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:23.860266 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.860234 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:23.926483 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:23.926417 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:24.126924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.126899 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz"] Oct 30 04:11:24.127793 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:11:24.127732 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda581c127_5a40_423e_8485_032317cce6ad.slice/crio-ce05d8b55525dc8d145aa66b75ebf3311825ce13b242a9ceb75a187cf9521123 WatchSource:0}: Error finding container ce05d8b55525dc8d145aa66b75ebf3311825ce13b242a9ceb75a187cf9521123: Status 404 returned error can't find the container with id ce05d8b55525dc8d145aa66b75ebf3311825ce13b242a9ceb75a187cf9521123 Oct 30 04:11:24.178675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.178648 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7"] Oct 30 04:11:24.179894 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:11:24.179858 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11b14d1_1342_447a_9494_d3b6d4a40de1.slice/crio-c34603bfb43f793357735d74907bf716bc57e6b4f11defb896894c8b043a9b10 WatchSource:0}: Error finding container c34603bfb43f793357735d74907bf716bc57e6b4f11defb896894c8b043a9b10: Status 404 returned error can't find the container with id c34603bfb43f793357735d74907bf716bc57e6b4f11defb896894c8b043a9b10 Oct 30 04:11:24.227815 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.227790 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6"] Oct 30 04:11:24.230190 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:11:24.230165 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c94d9d5_97ac_4930_9f88_7e83fbc37ef6.slice/crio-18f10ff6f1682a6839806152f7331c58296ffd4fdf699ccfd796d3a3bca96b2b WatchSource:0}: Error finding container 18f10ff6f1682a6839806152f7331c58296ffd4fdf699ccfd796d3a3bca96b2b: Status 404 returned error can't find the container with id 18f10ff6f1682a6839806152f7331c58296ffd4fdf699ccfd796d3a3bca96b2b Oct 30 04:11:24.231884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.231861 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:11:24.685983 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.685946 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:24.809672 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.809646 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" event={"ID":"b11b14d1-1342-447a-9494-d3b6d4a40de1","Type":"ContainerStarted","Data":"f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d"} Oct 30 04:11:24.809843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.809695 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" event={"ID":"b11b14d1-1342-447a-9494-d3b6d4a40de1","Type":"ContainerStarted","Data":"c34603bfb43f793357735d74907bf716bc57e6b4f11defb896894c8b043a9b10"} Oct 30 04:11:24.809843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.809764 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:24.811341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.811316 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" event={"ID":"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6","Type":"ContainerStarted","Data":"27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d"} Oct 30 04:11:24.811424 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.811360 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" event={"ID":"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6","Type":"ContainerStarted","Data":"18f10ff6f1682a6839806152f7331c58296ffd4fdf699ccfd796d3a3bca96b2b"} Oct 30 04:11:24.811489 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.811420 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:24.812896 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.812878 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" event={"ID":"a581c127-5a40-423e-8485-032317cce6ad","Type":"ContainerStarted","Data":"0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01"} Oct 30 04:11:24.812988 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.812909 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" event={"ID":"a581c127-5a40-423e-8485-032317cce6ad","Type":"ContainerStarted","Data":"ce05d8b55525dc8d145aa66b75ebf3311825ce13b242a9ceb75a187cf9521123"} Oct 30 04:11:24.812988 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.812929 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:24.844790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.844715 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" podStartSLOduration=1.8447032380000001 podStartE2EDuration="1.844703238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:24.843747357 +0000 UTC m=+4108.846100556" watchObservedRunningTime="2026-10-30 04:11:24.844703238 +0000 UTC m=+4108.847056435" Oct 30 04:11:24.859983 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:24.859955 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:11:24.883860 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.883817 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" podStartSLOduration=2.401495615 podStartE2EDuration="2.883804779s" totalImagesPullingTime="482.309164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:24.882261576 +0000 UTC m=+4108.884614774" watchObservedRunningTime="2026-10-30 04:11:24.883804779 +0000 UTC m=+4108.886158001" Oct 30 04:11:24.919125 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:24.919079 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" podStartSLOduration=1.9190671780000002 podStartE2EDuration="1.919067178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:11:24.917483977 +0000 UTC m=+4108.919837173" watchObservedRunningTime="2026-10-30 04:11:24.919067178 +0000 UTC m=+4108.921420375" Oct 30 04:11:26.100548 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:26.100514 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:26.742816 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:26.742777 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:11:27.506017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:27.505977 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:27.604344 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:27.604304 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:29.120954 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:29.120920 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:30.616697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:30.616663 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:30.823466 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:30.823437 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:32.216846 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:32.216817 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:33.701192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:33.701152 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:35.294482 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:35.294452 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:35.820910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:35.820879 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:35.821116 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:35.820974 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:36.797265 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.797229 4442 patch_prober.go:29] interesting pod/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 04:11:36.797748 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.797290 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 04:11:36.853904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.853870 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:36.869130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.869103 4442 generic.go:356] "Generic (PLEG): container finished" podID="b4554f90-1135-42c0-bb31-917f17d3c458" containerID="9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2" exitCode=0 Oct 30 04:11:36.869286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.869148 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" event={"ID":"b4554f90-1135-42c0-bb31-917f17d3c458","Type":"ContainerDied","Data":"9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2"} Oct 30 04:11:36.869533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.869519 4442 scope.go:122] "RemoveContainer" containerID="9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2" Oct 30 04:11:36.870798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.870772 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:11:36.870892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.870857 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:11:36.870941 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:36.870904 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:11:37.876512 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:37.876477 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" event={"ID":"b4554f90-1135-42c0-bb31-917f17d3c458","Type":"ContainerStarted","Data":"2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723"} Oct 30 04:11:38.320373 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:38.320342 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:38.856941 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:38.856908 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:38.857124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:38.857058 4442 patch_prober.go:29] interesting pod/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 04:11:38.857124 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:38.857107 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 04:11:39.856679 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:39.856647 4442 patch_prober.go:29] interesting pod/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 04:11:39.857141 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:39.856718 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 04:11:39.914256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:39.914223 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:40.857206 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:40.857169 4442 patch_prober.go:29] interesting pod/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 04:11:40.857622 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:40.857250 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 04:11:41.608187 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:41.608143 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:41.857016 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:41.856982 4442 patch_prober.go:29] interesting pod/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 04:11:41.857198 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:41.857038 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 04:11:42.882862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:42.882822 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:42.883972 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:42.883946 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:42.885495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:42.885470 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:43.093694 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:43.093661 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:44.579460 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:44.579421 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:45.997691 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:45.997653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:47.560624 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:47.560591 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:48.192978 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:48.192943 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz"] Oct 30 04:11:48.226905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:48.226872 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7"] Oct 30 04:11:48.322498 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:48.322420 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:48.421830 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:48.421800 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:48.999594 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:48.999562 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:50.156152 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.156122 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6"] Oct 30 04:11:50.156606 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.156325 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" podUID="7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" containerName="maas-api" containerID="cri-o://27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d" gracePeriod=30 Oct 30 04:11:50.201584 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.201556 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:50.415007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.414946 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:50.551249 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.551222 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-kube-api-access-79z8f\" (UniqueName: \"kubernetes.io/projected/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-kube-api-access-79z8f\") pod \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " Oct 30 04:11:50.551442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.551264 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls\") pod \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " Oct 30 04:11:50.551442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.551318 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-metrics-certs\") pod \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " Oct 30 04:11:50.551442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.551372 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-openshift-service-ca\") pod \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\" (UID: \"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6\") " Oct 30 04:11:50.551832 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.551805 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-openshift-service-ca" pod "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" (UID: "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:50.553502 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.553474 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-metrics-certs" pod "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" (UID: "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:50.553604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.553514 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls" pod "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" (UID: "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:50.553604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.553571 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-kube-api-access-79z8f" pod "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" (UID: "7c94d9d5-97ac-4930-9f88-7e83fbc37ef6"). InnerVolumeSpecName "kube-api-access-79z8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:50.626883 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.626822 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:50.652718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.652691 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79z8f\" (UniqueName: \"kubernetes.io/projected/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-kube-api-access-79z8f\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:50.652718 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.652716 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:50.652905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.652727 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:50.652905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.652735 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:50.728377 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.728298 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:50.967822 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.967786 4442 generic.go:356] "Generic (PLEG): container finished" podID="7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" containerID="27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d" exitCode=0 Oct 30 04:11:50.967998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.967834 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" Oct 30 04:11:50.967998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.967866 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" event={"ID":"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6","Type":"ContainerDied","Data":"27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d"} Oct 30 04:11:50.967998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.967922 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6" event={"ID":"7c94d9d5-97ac-4930-9f88-7e83fbc37ef6","Type":"ContainerDied","Data":"18f10ff6f1682a6839806152f7331c58296ffd4fdf699ccfd796d3a3bca96b2b"} Oct 30 04:11:50.967998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.967950 4442 scope.go:122] "RemoveContainer" containerID="27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d" Oct 30 04:11:50.977684 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.977663 4442 scope.go:122] "RemoveContainer" containerID="27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d" Oct 30 04:11:50.978083 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:50.978044 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d\": container with ID starting with 27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d not found: ID does not exist" containerID="27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d" Oct 30 04:11:50.978275 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:50.978080 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d"} err="failed to get container status \"27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d\": rpc error: code = NotFound desc = could not find container \"27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d\": container with ID starting with 27d357fe2ceabbc827c15858c60f7339ffe71cd9aaf66b8dbc7bf7aea547197d not found: ID does not exist" Oct 30 04:11:51.013430 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:51.013409 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6"] Oct 30 04:11:51.024439 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:51.024419 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6e1eb434-7f45d4fc9d-t29f6"] Oct 30 04:11:52.105730 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:52.105694 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" path="/var/lib/kubelet/pods/7c94d9d5-97ac-4930-9f88-7e83fbc37ef6/volumes" Oct 30 04:11:52.460646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:52.460544 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:11:53.016096 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.016057 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:53.193804 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.193745 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" podUID="a581c127-5a40-423e-8485-032317cce6ad" containerName="payload-processing" containerID="cri-o://0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01" gracePeriod=25 Oct 30 04:11:53.227481 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.227443 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" podUID="b11b14d1-1342-447a-9494-d3b6d4a40de1" containerName="payload-pre-processing" containerID="cri-o://f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d" gracePeriod=25 Oct 30 04:11:53.464248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.464220 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:53.492139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.492115 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:53.581327 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.581285 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11b14d1-1342-447a-9494-d3b6d4a40de1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11b14d1-1342-447a-9494-d3b6d4a40de1-plugins-config-volume\") pod \"b11b14d1-1342-447a-9494-d3b6d4a40de1\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " Oct 30 04:11:53.581501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.581406 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a581c127-5a40-423e-8485-032317cce6ad-kube-api-access-t92z5\" (UniqueName: \"kubernetes.io/projected/a581c127-5a40-423e-8485-032317cce6ad-kube-api-access-t92z5\") pod \"a581c127-5a40-423e-8485-032317cce6ad\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " Oct 30 04:11:53.581501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.581426 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11b14d1-1342-447a-9494-d3b6d4a40de1-kube-api-access-qrs2t\" (UniqueName: \"kubernetes.io/projected/b11b14d1-1342-447a-9494-d3b6d4a40de1-kube-api-access-qrs2t\") pod \"b11b14d1-1342-447a-9494-d3b6d4a40de1\" (UID: \"b11b14d1-1342-447a-9494-d3b6d4a40de1\") " Oct 30 04:11:53.581501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.581446 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a581c127-5a40-423e-8485-032317cce6ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a581c127-5a40-423e-8485-032317cce6ad-plugins-config-volume\") pod \"a581c127-5a40-423e-8485-032317cce6ad\" (UID: \"a581c127-5a40-423e-8485-032317cce6ad\") " Oct 30 04:11:53.581857 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.581713 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11b14d1-1342-447a-9494-d3b6d4a40de1-plugins-config-volume" pod "b11b14d1-1342-447a-9494-d3b6d4a40de1" (UID: "b11b14d1-1342-447a-9494-d3b6d4a40de1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:53.582007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.581933 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a581c127-5a40-423e-8485-032317cce6ad-plugins-config-volume" pod "a581c127-5a40-423e-8485-032317cce6ad" (UID: "a581c127-5a40-423e-8485-032317cce6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:53.583628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.583588 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a581c127-5a40-423e-8485-032317cce6ad-kube-api-access-t92z5" pod "a581c127-5a40-423e-8485-032317cce6ad" (UID: "a581c127-5a40-423e-8485-032317cce6ad"). InnerVolumeSpecName "kube-api-access-t92z5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:53.583628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.583613 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11b14d1-1342-447a-9494-d3b6d4a40de1-kube-api-access-qrs2t" pod "b11b14d1-1342-447a-9494-d3b6d4a40de1" (UID: "b11b14d1-1342-447a-9494-d3b6d4a40de1"). InnerVolumeSpecName "kube-api-access-qrs2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:53.683067 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.683039 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t92z5\" (UniqueName: \"kubernetes.io/projected/a581c127-5a40-423e-8485-032317cce6ad-kube-api-access-t92z5\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:53.683067 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.683061 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrs2t\" (UniqueName: \"kubernetes.io/projected/b11b14d1-1342-447a-9494-d3b6d4a40de1-kube-api-access-qrs2t\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:53.683067 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.683072 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a581c127-5a40-423e-8485-032317cce6ad-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:53.683263 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.683080 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b11b14d1-1342-447a-9494-d3b6d4a40de1-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:53.983464 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.983380 4442 generic.go:356] "Generic (PLEG): container finished" podID="b11b14d1-1342-447a-9494-d3b6d4a40de1" containerID="f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d" exitCode=0 Oct 30 04:11:53.983464 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.983434 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" Oct 30 04:11:53.983697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.983469 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" event={"ID":"b11b14d1-1342-447a-9494-d3b6d4a40de1","Type":"ContainerDied","Data":"f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d"} Oct 30 04:11:53.983697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.983525 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7" event={"ID":"b11b14d1-1342-447a-9494-d3b6d4a40de1","Type":"ContainerDied","Data":"c34603bfb43f793357735d74907bf716bc57e6b4f11defb896894c8b043a9b10"} Oct 30 04:11:53.983697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.983547 4442 scope.go:122] "RemoveContainer" containerID="f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d" Oct 30 04:11:53.984859 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.984834 4442 generic.go:356] "Generic (PLEG): container finished" podID="a581c127-5a40-423e-8485-032317cce6ad" containerID="0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01" exitCode=0 Oct 30 04:11:53.984974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.984879 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" event={"ID":"a581c127-5a40-423e-8485-032317cce6ad","Type":"ContainerDied","Data":"0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01"} Oct 30 04:11:53.984974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.984938 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" event={"ID":"a581c127-5a40-423e-8485-032317cce6ad","Type":"ContainerDied","Data":"ce05d8b55525dc8d145aa66b75ebf3311825ce13b242a9ceb75a187cf9521123"} Oct 30 04:11:53.984974 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.984885 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz" Oct 30 04:11:53.993735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.993709 4442 scope.go:122] "RemoveContainer" containerID="f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d" Oct 30 04:11:53.994047 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:53.994025 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d\": container with ID starting with f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d not found: ID does not exist" containerID="f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d" Oct 30 04:11:53.994138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.994060 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d"} err="failed to get container status \"f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d\": rpc error: code = NotFound desc = could not find container \"f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d\": container with ID starting with f13f36e9e44d954eb9cc081a97548bc72afe624501c49357276a933e4753103d not found: ID does not exist" Oct 30 04:11:53.994138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:53.994084 4442 scope.go:122] "RemoveContainer" containerID="0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01" Oct 30 04:11:54.003005 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.002988 4442 scope.go:122] "RemoveContainer" containerID="0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01" Oct 30 04:11:54.003308 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:54.003284 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01\": container with ID starting with 0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01 not found: ID does not exist" containerID="0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01" Oct 30 04:11:54.003419 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.003312 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01"} err="failed to get container status \"0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01\": rpc error: code = NotFound desc = could not find container \"0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01\": container with ID starting with 0b9c705d004badc2a0c1e57dacd061a1b012adb4854d5d4791ca5f617df8eb01 not found: ID does not exist" Oct 30 04:11:54.016213 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.016194 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:54.029047 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.029025 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7"] Oct 30 04:11:54.039881 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.039859 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6e1eb434-7b779bbccf-cxnq7"] Oct 30 04:11:54.063119 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.063093 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz"] Oct 30 04:11:54.067178 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.067158 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6e1eb434-c74ff4f98-4xflz"] Oct 30 04:11:54.105620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.105593 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a581c127-5a40-423e-8485-032317cce6ad" path="/var/lib/kubelet/pods/a581c127-5a40-423e-8485-032317cce6ad/volumes" Oct 30 04:11:54.106033 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.106019 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11b14d1-1342-447a-9494-d3b6d4a40de1" path="/var/lib/kubelet/pods/b11b14d1-1342-447a-9494-d3b6d4a40de1/volumes" Oct 30 04:11:54.193414 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.193369 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj"] Oct 30 04:11:54.193639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.193550 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" containerID="cri-o://2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723" gracePeriod=30 Oct 30 04:11:54.216397 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:54.216325 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:54.860146 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:54.860111 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:11:55.787668 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:55.787626 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:57.886919 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:57.886879 4442 patch_prober.go:29] interesting pod/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:11:57.887329 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:57.886952 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:11:58.192591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.192494 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4dk5t"] Oct 30 04:11:58.192823 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.192713 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-679796c8cb-4dk5t" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" containerName="manager" containerID="cri-o://98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94" gracePeriod=30 Oct 30 04:11:58.437401 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.437373 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 04:11:58.527349 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.527318 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-metrics-certs\") pod \"05381e85-4985-4f92-9d08-6cca60bf8059\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " Oct 30 04:11:58.527542 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.527395 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-webhook-cert\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-webhook-cert\") pod \"05381e85-4985-4f92-9d08-6cca60bf8059\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " Oct 30 04:11:58.527542 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.527435 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05381e85-4985-4f92-9d08-6cca60bf8059-kube-api-access-qxbz5\" (UniqueName: \"kubernetes.io/projected/05381e85-4985-4f92-9d08-6cca60bf8059-kube-api-access-qxbz5\") pod \"05381e85-4985-4f92-9d08-6cca60bf8059\" (UID: \"05381e85-4985-4f92-9d08-6cca60bf8059\") " Oct 30 04:11:58.529581 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.529549 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-metrics-certs" pod "05381e85-4985-4f92-9d08-6cca60bf8059" (UID: "05381e85-4985-4f92-9d08-6cca60bf8059"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:58.529694 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.529660 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05381e85-4985-4f92-9d08-6cca60bf8059-kube-api-access-qxbz5" pod "05381e85-4985-4f92-9d08-6cca60bf8059" (UID: "05381e85-4985-4f92-9d08-6cca60bf8059"). InnerVolumeSpecName "kube-api-access-qxbz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:58.529738 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.529724 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-webhook-cert" pod "05381e85-4985-4f92-9d08-6cca60bf8059" (UID: "05381e85-4985-4f92-9d08-6cca60bf8059"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:11:58.628563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.628522 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:58.628563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.628555 4442 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05381e85-4985-4f92-9d08-6cca60bf8059-webhook-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:58.628563 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:58.628568 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxbz5\" (UniqueName: \"kubernetes.io/projected/05381e85-4985-4f92-9d08-6cca60bf8059-kube-api-access-qxbz5\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.012326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.012293 4442 generic.go:356] "Generic (PLEG): container finished" podID="05381e85-4985-4f92-9d08-6cca60bf8059" containerID="98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94" exitCode=0 Oct 30 04:11:59.012904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.012336 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4dk5t" event={"ID":"05381e85-4985-4f92-9d08-6cca60bf8059","Type":"ContainerDied","Data":"98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94"} Oct 30 04:11:59.012904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.012343 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4dk5t" Oct 30 04:11:59.012904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.012389 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4dk5t" event={"ID":"05381e85-4985-4f92-9d08-6cca60bf8059","Type":"ContainerDied","Data":"fe1202396429b457c4b41fd85cbf7bb733c24d62756339eef05c2600c7ac71e5"} Oct 30 04:11:59.012904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.012421 4442 scope.go:122] "RemoveContainer" containerID="98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94" Oct 30 04:11:59.024700 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.024681 4442 scope.go:122] "RemoveContainer" containerID="3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911" Oct 30 04:11:59.033984 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.033959 4442 scope.go:122] "RemoveContainer" containerID="98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94" Oct 30 04:11:59.034289 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:59.034273 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94\": container with ID starting with 98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94 not found: ID does not exist" containerID="98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94" Oct 30 04:11:59.034358 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.034296 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94"} err="failed to get container status \"98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94\": rpc error: code = NotFound desc = could not find container \"98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94\": container with ID starting with 98de8385a3f897a3c07e6872abaa48c8847a5430403678dec8e1fa7970bd1b94 not found: ID does not exist" Oct 30 04:11:59.034358 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.034311 4442 scope.go:122] "RemoveContainer" containerID="3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911" Oct 30 04:11:59.034524 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:59.034510 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911\": container with ID starting with 3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911 not found: ID does not exist" containerID="3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911" Oct 30 04:11:59.034570 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.034528 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911"} err="failed to get container status \"3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911\": rpc error: code = NotFound desc = could not find container \"3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911\": container with ID starting with 3877f9275711794e5365928cabcda0de76c0af467536976c6f86ffa918523911 not found: ID does not exist" Oct 30 04:11:59.059590 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.059561 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4dk5t"] Oct 30 04:11:59.070527 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.070505 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4dk5t"] Oct 30 04:11:59.304420 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:11:59.304344 4442 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4554f90_1135_42c0_bb31_917f17d3c458.slice/crio-878e9972240228e382e51a7b73a6ceb79b2647c4cf3139d8ef21a09ae1aabc5b\": RecentStats: unable to find data in memory cache]" Oct 30 04:11:59.462173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.462138 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:11:59.637567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637479 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-credential-socket\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.637567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637527 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-istio-token\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-istio-token\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.637567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637566 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-certs\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637600 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4554f90-1135-42c0-bb31-917f17d3c458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4554f90-1135-42c0-bb31-917f17d3c458-istio-podinfo\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637652 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-data\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637721 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-socket\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637765 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4554f90-1135-42c0-bb31-917f17d3c458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4554f90-1135-42c0-bb31-917f17d3c458-istiod-ca-cert\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637792 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-envoy\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637832 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-kube-api-access-rvdwj\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-kube-api-access-rvdwj\") pod \"b4554f90-1135-42c0-bb31-917f17d3c458\" (UID: \"b4554f90-1135-42c0-bb31-917f17d3c458\") " Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637886 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-credential-socket" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.637972 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-certs" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.638068 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-socket" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:59.638092 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.638081 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-data" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:59.638770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.638202 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.638770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.638221 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.638770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.638268 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.638770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.638282 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.638770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.638385 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4554f90-1135-42c0-bb31-917f17d3c458-istiod-ca-cert" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:11:59.640131 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.640108 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-istio-token" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:59.640276 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.640253 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4554f90-1135-42c0-bb31-917f17d3c458-istio-podinfo" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:11:59.640613 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.640593 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-kube-api-access-rvdwj" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "kube-api-access-rvdwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:11:59.640706 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.640643 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-envoy" pod "b4554f90-1135-42c0-bb31-917f17d3c458" (UID: "b4554f90-1135-42c0-bb31-917f17d3c458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:11:59.739171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.739122 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdwj\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-kube-api-access-rvdwj\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.739171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.739158 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4554f90-1135-42c0-bb31-917f17d3c458-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.739171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.739167 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4554f90-1135-42c0-bb31-917f17d3c458-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.739171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.739175 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4554f90-1135-42c0-bb31-917f17d3c458-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.739171 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.739184 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4554f90-1135-42c0-bb31-917f17d3c458-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:11:59.849078 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849044 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-679796c8cb-4z5zf"] Oct 30 04:11:59.849400 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849388 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" Oct 30 04:11:59.849451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849402 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" Oct 30 04:11:59.849451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849418 4442 cpu_manager.go:414] "containerMap: removing container" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" containerName="manager" Oct 30 04:11:59.849451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849423 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" containerName="manager" Oct 30 04:11:59.849451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849435 4442 cpu_manager.go:414] "containerMap: removing container" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" containerName="manager" Oct 30 04:11:59.849451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849440 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" containerName="manager" Oct 30 04:11:59.849451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849446 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b11b14d1-1342-447a-9494-d3b6d4a40de1" containerName="payload-pre-processing" Oct 30 04:11:59.849451 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849451 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11b14d1-1342-447a-9494-d3b6d4a40de1" containerName="payload-pre-processing" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849468 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" containerName="maas-api" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849473 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" containerName="maas-api" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849481 4442 cpu_manager.go:414] "containerMap: removing container" podUID="a581c127-5a40-423e-8485-032317cce6ad" containerName="payload-processing" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849485 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a581c127-5a40-423e-8485-032317cce6ad" containerName="payload-processing" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849534 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c94d9d5-97ac-4930-9f88-7e83fbc37ef6" containerName="maas-api" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849542 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="a581c127-5a40-423e-8485-032317cce6ad" containerName="payload-processing" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849547 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849554 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849561 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" containerName="manager" Oct 30 04:11:59.849703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.849568 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11b14d1-1342-447a-9494-d3b6d4a40de1" containerName="payload-pre-processing" Oct 30 04:11:59.853056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.853036 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:11:59.866634 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.866611 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8sjhl\"" Oct 30 04:11:59.866742 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.866653 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 04:11:59.866742 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.866677 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 04:11:59.877018 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.876997 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 04:11:59.879531 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:11:59.879508 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4z5zf"] Oct 30 04:12:00.021659 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.021619 4442 generic.go:356] "Generic (PLEG): container finished" podID="b4554f90-1135-42c0-bb31-917f17d3c458" containerID="2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723" exitCode=0 Oct 30 04:12:00.022220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.021667 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" Oct 30 04:12:00.022220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.021699 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" event={"ID":"b4554f90-1135-42c0-bb31-917f17d3c458","Type":"ContainerDied","Data":"2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723"} Oct 30 04:12:00.022220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.021783 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj" event={"ID":"b4554f90-1135-42c0-bb31-917f17d3c458","Type":"ContainerDied","Data":"878e9972240228e382e51a7b73a6ceb79b2647c4cf3139d8ef21a09ae1aabc5b"} Oct 30 04:12:00.022220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.021825 4442 scope.go:122] "RemoveContainer" containerID="2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723" Oct 30 04:12:00.036271 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.036254 4442 scope.go:122] "RemoveContainer" containerID="9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2" Oct 30 04:12:00.042028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.041999 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkt8b\" (UniqueName: \"kubernetes.io/projected/44c3d906-08b1-42b4-aa9b-acb7440a4e12-kube-api-access-bkt8b\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.042166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.042041 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-webhook-cert\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.042240 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.042162 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-metrics-certs\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.044666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.044642 4442 scope.go:122] "RemoveContainer" containerID="2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723" Oct 30 04:12:00.044992 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:12:00.044973 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723\": container with ID starting with 2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723 not found: ID does not exist" containerID="2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723" Oct 30 04:12:00.045138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.045004 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723"} err="failed to get container status \"2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723\": rpc error: code = NotFound desc = could not find container \"2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723\": container with ID starting with 2b0d046bc4a2eee22ccff6318c8c5b4fa8afbae4fb32e438deae4551d837b723 not found: ID does not exist" Oct 30 04:12:00.045138 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.045027 4442 scope.go:122] "RemoveContainer" containerID="9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2" Oct 30 04:12:00.045354 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:12:00.045336 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2\": container with ID starting with 9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2 not found: ID does not exist" containerID="9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2" Oct 30 04:12:00.045415 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.045364 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2"} err="failed to get container status \"9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2\": rpc error: code = NotFound desc = could not find container \"9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2\": container with ID starting with 9e5d0b827d68e36ba0dfdbb8ee3adb4a40993a21243768b493ae16a48fee15f2 not found: ID does not exist" Oct 30 04:12:00.069556 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.069534 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj"] Oct 30 04:12:00.073139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.073117 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6e1eb434-openshift-default-64b95b4ff7-gc4vj"] Oct 30 04:12:00.107029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.106997 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" path="/var/lib/kubelet/pods/05381e85-4985-4f92-9d08-6cca60bf8059/volumes" Oct 30 04:12:00.107669 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.107649 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" path="/var/lib/kubelet/pods/b4554f90-1135-42c0-bb31-917f17d3c458/volumes" Oct 30 04:12:00.143185 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.143152 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkt8b\" (UniqueName: \"kubernetes.io/projected/44c3d906-08b1-42b4-aa9b-acb7440a4e12-kube-api-access-bkt8b\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.143326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.143202 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-webhook-cert\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.143326 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.143265 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-metrics-certs\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.146611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.146583 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-metrics-certs\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.146713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.146621 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-webhook-cert\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.160977 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.160956 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkt8b\" (UniqueName: \"kubernetes.io/projected/44c3d906-08b1-42b4-aa9b-acb7440a4e12-kube-api-access-bkt8b\") pod \"maas-controller-679796c8cb-4z5zf\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.165715 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.165699 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:00.467111 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:00.467087 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4z5zf"] Oct 30 04:12:00.469210 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:12:00.469180 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c3d906_08b1_42b4_aa9b_acb7440a4e12.slice/crio-5c390be9d9e5b72d429fd279ea5beacca508dd94ad4046bebe6927a9a5378552 WatchSource:0}: Error finding container 5c390be9d9e5b72d429fd279ea5beacca508dd94ad4046bebe6927a9a5378552: Status 404 returned error can't find the container with id 5c390be9d9e5b72d429fd279ea5beacca508dd94ad4046bebe6927a9a5378552 Oct 30 04:12:01.031896 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:01.031857 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4z5zf" event={"ID":"44c3d906-08b1-42b4-aa9b-acb7440a4e12","Type":"ContainerStarted","Data":"929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01"} Oct 30 04:12:01.032304 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:01.031915 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4z5zf" event={"ID":"44c3d906-08b1-42b4-aa9b-acb7440a4e12","Type":"ContainerStarted","Data":"5c390be9d9e5b72d429fd279ea5beacca508dd94ad4046bebe6927a9a5378552"} Oct 30 04:12:01.032304 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:01.031957 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:01.070422 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:01.070376 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-679796c8cb-4z5zf" podStartSLOduration=1.611090675 podStartE2EDuration="2.070360722s" totalImagesPullingTime="459.270047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:01.069142157 +0000 UTC m=+4145.071495353" watchObservedRunningTime="2026-10-30 04:12:01.070360722 +0000 UTC m=+4145.072713919" Oct 30 04:12:12.039376 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.039347 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:12.275195 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.275160 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d"] Oct 30 04:12:12.275501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.275488 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" Oct 30 04:12:12.275549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.275502 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4554f90-1135-42c0-bb31-917f17d3c458" containerName="istio-proxy" Oct 30 04:12:12.275590 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.275569 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="05381e85-4985-4f92-9d08-6cca60bf8059" containerName="manager" Oct 30 04:12:12.278567 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.278549 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.284661 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.284622 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm"] Oct 30 04:12:12.287931 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.287910 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q"] Oct 30 04:12:12.288056 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.288028 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.291319 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.291270 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.297101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.297072 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d"] Oct 30 04:12:12.307958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.306721 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm"] Oct 30 04:12:12.308821 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.308795 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q"] Oct 30 04:12:12.339467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339443 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.339598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339473 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.339598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339492 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szhjb\" (UniqueName: \"kubernetes.io/projected/8894e5e4-3e55-4cae-9400-27794bd21f0a-kube-api-access-szhjb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.339598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339519 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.339598 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339566 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjnw\" (UniqueName: \"kubernetes.io/projected/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-kube-api-access-5kjnw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.339798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339614 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.339798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339670 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.339798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339687 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.339798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339705 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.339798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339728 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.339798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339783 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.340028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339838 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.340028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339860 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.340028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339895 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.340028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339915 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.340028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339933 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.340028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339949 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.340028 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.339999 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.340277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340028 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.340277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340107 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.340277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340125 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.340277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340147 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.340277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340189 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.340277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340215 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.340277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340246 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8l7\" (UniqueName: \"kubernetes.io/projected/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-kube-api-access-jt8l7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.340529 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340285 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.340529 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.340302 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8894e5e4-3e55-4cae-9400-27794bd21f0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.441192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441165 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.441192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441198 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.441442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441214 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.441442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441236 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.441442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441255 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.441442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441384 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8l7\" (UniqueName: \"kubernetes.io/projected/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-kube-api-access-jt8l7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.441442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441442 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441471 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8894e5e4-3e55-4cae-9400-27794bd21f0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441506 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441533 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441556 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szhjb\" (UniqueName: \"kubernetes.io/projected/8894e5e4-3e55-4cae-9400-27794bd21f0a-kube-api-access-szhjb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441597 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441628 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjnw\" (UniqueName: \"kubernetes.io/projected/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-kube-api-access-5kjnw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441656 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.441722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441700 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441727 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441748 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441786 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441872 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441908 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441941 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.441982 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442011 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442046 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442080 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442108 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442134 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.442153 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442155 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442307 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442400 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442449 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442499 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442624 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8894e5e4-3e55-4cae-9400-27794bd21f0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442715 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442776 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442919 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.443011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.442960 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.443520 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.443189 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-workload-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.443520 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.443463 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.443656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.443626 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-data\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.443733 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.443711 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-workload-certs\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.445051 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.444254 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-credential-socket\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.445051 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.444685 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.445051 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.444929 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.445301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.445228 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.445515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.445494 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.445665 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.445644 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.445938 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.445920 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.459798 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.459770 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.460175 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.460145 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szhjb\" (UniqueName: \"kubernetes.io/projected/8894e5e4-3e55-4cae-9400-27794bd21f0a-kube-api-access-szhjb\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.460263 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.460151 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8894e5e4-3e55-4cae-9400-27794bd21f0a-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-bp56d\" (UID: \"8894e5e4-3e55-4cae-9400-27794bd21f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.460520 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.460497 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8l7\" (UniqueName: \"kubernetes.io/projected/2c23bba8-edb0-4f3d-9d05-aca46b96e13c-kube-api-access-jt8l7\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-sq5vm\" (UID: \"2c23bba8-edb0-4f3d-9d05-aca46b96e13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.460661 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.460642 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjnw\" (UniqueName: \"kubernetes.io/projected/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-kube-api-access-5kjnw\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.460814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.460797 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908-istio-token\") pod \"maas-default-gateway-openshift-default-77bfb58f8c-s6z4q\" (UID: \"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.593270 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.593200 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:12.604583 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.604564 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:12.614658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.614634 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:12.910769 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.910722 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d"] Oct 30 04:12:12.912900 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.912870 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm"] Oct 30 04:12:12.914425 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.914379 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.914535 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.914514 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.914627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.914606 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.922886 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:12:12.922865 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c23bba8_edb0_4f3d_9d05_aca46b96e13c.slice/crio-8565e996c47bb3d09669451a7de1ec9e4527dfb671095196988633953d1d8f35 WatchSource:0}: Error finding container 8565e996c47bb3d09669451a7de1ec9e4527dfb671095196988633953d1d8f35: Status 404 returned error can't find the container with id 8565e996c47bb3d09669451a7de1ec9e4527dfb671095196988633953d1d8f35 Oct 30 04:12:12.925226 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.925190 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.925311 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.925252 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.925311 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.925278 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.932840 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.932813 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q"] Oct 30 04:12:12.938580 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.938551 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.938663 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.938611 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:12.938663 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:12.938653 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:12:13.087497 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.086739 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" event={"ID":"8894e5e4-3e55-4cae-9400-27794bd21f0a","Type":"ContainerStarted","Data":"6a6c2c78a532665b04b2652dd3481da22247697a2d4841bbf0ba3bcee0976c38"} Oct 30 04:12:13.087497 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.086813 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" event={"ID":"8894e5e4-3e55-4cae-9400-27794bd21f0a","Type":"ContainerStarted","Data":"2f28d3748700e1e09923729b987188aae3e29bdab98d94ee873560091d0a1d84"} Oct 30 04:12:13.092021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.091301 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" event={"ID":"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908","Type":"ContainerStarted","Data":"ff6e7480b8a4405bce6f3c18bda37a6c3c9120223da1f50c49b157b97e981c3f"} Oct 30 04:12:13.092021 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.091354 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" event={"ID":"2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908","Type":"ContainerStarted","Data":"94da81658c54cef78640b5d05ff84bfeb04f6e8f633768339c3eea46d7dcf49e"} Oct 30 04:12:13.093825 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.093442 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" event={"ID":"2c23bba8-edb0-4f3d-9d05-aca46b96e13c","Type":"ContainerStarted","Data":"c0a870690ea2f3e2198e60394abf1c32f0c62b74245ae26ee38abd6709766390"} Oct 30 04:12:13.093825 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.093480 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" event={"ID":"2c23bba8-edb0-4f3d-9d05-aca46b96e13c","Type":"ContainerStarted","Data":"8565e996c47bb3d09669451a7de1ec9e4527dfb671095196988633953d1d8f35"} Oct 30 04:12:13.127794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.127725 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podStartSLOduration=1.127711701 podStartE2EDuration="1.127711701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:13.124328961 +0000 UTC m=+4157.126682171" watchObservedRunningTime="2026-10-30 04:12:13.127711701 +0000 UTC m=+4157.130064897" Oct 30 04:12:13.162859 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.162742 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podStartSLOduration=1.16272843 podStartE2EDuration="1.16272843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:13.161243587 +0000 UTC m=+4157.163596786" watchObservedRunningTime="2026-10-30 04:12:13.16272843 +0000 UTC m=+4157.165081626" Oct 30 04:12:13.203670 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.203607 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podStartSLOduration=1.20358715 podStartE2EDuration="1.20358715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:13.201919495 +0000 UTC m=+4157.204272726" watchObservedRunningTime="2026-10-30 04:12:13.20358715 +0000 UTC m=+4157.205940348" Oct 30 04:12:13.595814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.595776 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:13.596202 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.596177 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:13.596370 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.596346 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:13.607491 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.607451 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:13.607801 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.607744 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:13.608000 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.607805 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:13.617379 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.617351 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:13.617703 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.617680 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:13.617905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:13.617880 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:14.594274 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:14.594230 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:14.594952 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:14.594919 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:14.607163 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:14.607130 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:14.607376 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:14.607353 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:14.618883 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:14.618855 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:14.619183 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:14.619159 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:15.594791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:15.594194 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:15.594791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:15.594261 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:15.608782 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:15.606114 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:15.608782 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:15.606171 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:15.619685 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:15.619652 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:15.619934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:15.619909 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:16.599328 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:16.599293 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:16.600014 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:16.599961 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:16.612146 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:16.612116 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:16.612367 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:16.612341 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:16.617077 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:16.617052 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:16.617261 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:16.617240 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:17.594166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:17.594127 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:17.594890 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:17.594848 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:17.606126 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:17.606094 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:17.606667 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:17.606621 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:17.625776 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:17.618157 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:17.625776 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:17.618205 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:18.597778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:18.597143 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:18.597778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:18.597207 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:18.613777 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:18.612878 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:18.613777 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:18.612936 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:18.617774 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:18.617595 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:18.617774 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:18.617644 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:19.593730 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:19.593690 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:19.593924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:19.593774 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:19.605075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:19.605038 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:19.605225 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:19.605076 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:19.615303 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:19.615277 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:19.615838 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:19.615806 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:20.593770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:20.593725 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" start-of-body= Oct 30 04:12:20.593993 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:20.593811 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": dial tcp 10.134.0.247:15021: connect: connection refused" Oct 30 04:12:20.605286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:20.605243 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" start-of-body= Oct 30 04:12:20.605426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:20.605304 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": dial tcp 10.134.0.248:15021: connect: connection refused" Oct 30 04:12:20.615490 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:20.615451 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" start-of-body= Oct 30 04:12:20.615951 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:20.615500 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": dial tcp 10.134.0.249:15021: connect: connection refused" Oct 30 04:12:22.605386 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:22.605346 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:12:22.606063 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:22.606030 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:12:22.615898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:22.615872 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:12:22.616103 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:22.616072 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:12:22.635915 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:22.635878 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:12:22.636086 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:22.636063 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:12:23.652924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:23.652870 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:12:23.653571 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:23.653503 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:12:23.657923 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:23.657899 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:12:23.658131 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:23.658106 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:12:23.680886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:23.680859 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:12:23.680991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:23.680899 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:12:24.129915 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.129884 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:24.131103 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.131076 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:24.135963 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.135840 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" Oct 30 04:12:24.399800 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.399701 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:24.400053 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.400027 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:24.400796 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.400779 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" Oct 30 04:12:24.418820 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.418795 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:24.419199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.419176 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:24.419892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:24.419876 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" Oct 30 04:12:24.859982 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:12:24.859937 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:12:29.715277 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:29.715241 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:29.878022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:29.877986 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4z5zf"] Oct 30 04:12:29.878262 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:29.878241 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-679796c8cb-4z5zf" podUID="44c3d906-08b1-42b4-aa9b-acb7440a4e12" containerName="manager" containerID="cri-o://929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01" gracePeriod=30 Oct 30 04:12:30.128711 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.128685 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:30.219301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.219263 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c3d906-08b1-42b4-aa9b-acb7440a4e12-kube-api-access-bkt8b\" (UniqueName: \"kubernetes.io/projected/44c3d906-08b1-42b4-aa9b-acb7440a4e12-kube-api-access-bkt8b\") pod \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " Oct 30 04:12:30.219485 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.219383 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-webhook-cert\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-webhook-cert\") pod \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " Oct 30 04:12:30.219485 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.219466 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-metrics-certs\") pod \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\" (UID: \"44c3d906-08b1-42b4-aa9b-acb7440a4e12\") " Oct 30 04:12:30.221572 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.221533 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-webhook-cert" pod "44c3d906-08b1-42b4-aa9b-acb7440a4e12" (UID: "44c3d906-08b1-42b4-aa9b-acb7440a4e12"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:30.221713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.221542 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c3d906-08b1-42b4-aa9b-acb7440a4e12-kube-api-access-bkt8b" pod "44c3d906-08b1-42b4-aa9b-acb7440a4e12" (UID: "44c3d906-08b1-42b4-aa9b-acb7440a4e12"). InnerVolumeSpecName "kube-api-access-bkt8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:30.221713 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.221606 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-metrics-certs" pod "44c3d906-08b1-42b4-aa9b-acb7440a4e12" (UID: "44c3d906-08b1-42b4-aa9b-acb7440a4e12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:12:30.284689 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.284660 4442 generic.go:356] "Generic (PLEG): container finished" podID="44c3d906-08b1-42b4-aa9b-acb7440a4e12" containerID="929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01" exitCode=0 Oct 30 04:12:30.284892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.284699 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-4z5zf" Oct 30 04:12:30.284892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.284741 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4z5zf" event={"ID":"44c3d906-08b1-42b4-aa9b-acb7440a4e12","Type":"ContainerDied","Data":"929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01"} Oct 30 04:12:30.284892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.284817 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-4z5zf" event={"ID":"44c3d906-08b1-42b4-aa9b-acb7440a4e12","Type":"ContainerDied","Data":"5c390be9d9e5b72d429fd279ea5beacca508dd94ad4046bebe6927a9a5378552"} Oct 30 04:12:30.284892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.284841 4442 scope.go:122] "RemoveContainer" containerID="929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01" Oct 30 04:12:30.293656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.293640 4442 scope.go:122] "RemoveContainer" containerID="929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01" Oct 30 04:12:30.293954 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:12:30.293933 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01\": container with ID starting with 929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01 not found: ID does not exist" containerID="929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01" Oct 30 04:12:30.294042 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.293963 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01"} err="failed to get container status \"929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01\": rpc error: code = NotFound desc = could not find container \"929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01\": container with ID starting with 929dd9fbf81f3d1b16587dfac40044b7e2dd01b5ad9da5f8df347071d8ea9e01 not found: ID does not exist" Oct 30 04:12:30.320946 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.320918 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkt8b\" (UniqueName: \"kubernetes.io/projected/44c3d906-08b1-42b4-aa9b-acb7440a4e12-kube-api-access-bkt8b\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:12:30.320946 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.320944 4442 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-webhook-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:12:30.321081 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.320958 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44c3d906-08b1-42b4-aa9b-acb7440a4e12-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:12:30.329978 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.329954 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4z5zf"] Oct 30 04:12:30.335235 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:30.335214 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-679796c8cb-4z5zf"] Oct 30 04:12:31.373984 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.373945 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:31.538449 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.538410 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-679796c8cb-rwz7m"] Oct 30 04:12:31.538881 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.538862 4442 cpu_manager.go:414] "containerMap: removing container" podUID="44c3d906-08b1-42b4-aa9b-acb7440a4e12" containerName="manager" Oct 30 04:12:31.538989 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.538882 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c3d906-08b1-42b4-aa9b-acb7440a4e12" containerName="manager" Oct 30 04:12:31.539060 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.539015 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c3d906-08b1-42b4-aa9b-acb7440a4e12" containerName="manager" Oct 30 04:12:31.543349 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.543328 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.557221 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.557200 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 04:12:31.567917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.567894 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8sjhl\"" Oct 30 04:12:31.568039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.567966 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 04:12:31.568039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.567966 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 04:12:31.570441 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.570419 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679796c8cb-rwz7m"] Oct 30 04:12:31.630680 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.630603 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpjj\" (UniqueName: \"kubernetes.io/projected/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-kube-api-access-vfpjj\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.630843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.630719 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-webhook-cert\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.630892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.630852 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-metrics-certs\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.731896 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.731866 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-metrics-certs\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.732039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.731946 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpjj\" (UniqueName: \"kubernetes.io/projected/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-kube-api-access-vfpjj\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.732039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.731983 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-webhook-cert\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.734569 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.734546 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-webhook-cert\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.734668 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.734613 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-metrics-certs\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.749737 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.749706 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpjj\" (UniqueName: \"kubernetes.io/projected/d83c13ee-a5c0-4a5f-8a33-57aeb805a233-kube-api-access-vfpjj\") pod \"maas-controller-679796c8cb-rwz7m\" (UID: \"d83c13ee-a5c0-4a5f-8a33-57aeb805a233\") " pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:31.856134 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:31.856099 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:32.012690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:32.012624 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:32.087276 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:32.087250 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:32.104928 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:32.104902 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c3d906-08b1-42b4-aa9b-acb7440a4e12" path="/var/lib/kubelet/pods/44c3d906-08b1-42b4-aa9b-acb7440a4e12/volumes" Oct 30 04:12:32.149496 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:32.149472 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-679796c8cb-rwz7m"] Oct 30 04:12:32.151090 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:12:32.151059 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83c13ee_a5c0_4a5f_8a33_57aeb805a233.slice/crio-e27ebd93f19959d7efa6327e37f0671eb71adf69f9094c9f4669d65b60152fa6 WatchSource:0}: Error finding container e27ebd93f19959d7efa6327e37f0671eb71adf69f9094c9f4669d65b60152fa6: Status 404 returned error can't find the container with id e27ebd93f19959d7efa6327e37f0671eb71adf69f9094c9f4669d65b60152fa6 Oct 30 04:12:32.295600 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:32.295538 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-rwz7m" event={"ID":"d83c13ee-a5c0-4a5f-8a33-57aeb805a233","Type":"ContainerStarted","Data":"e27ebd93f19959d7efa6327e37f0671eb71adf69f9094c9f4669d65b60152fa6"} Oct 30 04:12:33.301068 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:33.301030 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-679796c8cb-rwz7m" event={"ID":"d83c13ee-a5c0-4a5f-8a33-57aeb805a233","Type":"ContainerStarted","Data":"f2d289ee75d208c4f7ae5ffd82aef59201ec8bd1965e49c236fee7b4c95d4e94"} Oct 30 04:12:33.301480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:33.301178 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:33.344315 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:33.339455 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-679796c8cb-rwz7m" podStartSLOduration=1.925890201 podStartE2EDuration="2.339437663s" totalImagesPullingTime="413.547462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:33.337649135 +0000 UTC m=+4177.340002333" watchObservedRunningTime="2026-10-30 04:12:33.339437663 +0000 UTC m=+4177.341790862" Oct 30 04:12:35.031246 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:35.031207 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:35.135136 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:35.135107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:39.160576 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:39.160497 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:39.246998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:39.246967 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:44.308160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:44.308131 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-679796c8cb-rwz7m" Oct 30 04:12:53.147402 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:53.147368 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:54.131163 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:54.131131 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:54.458004 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:54.457929 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:12:54.867478 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:12:54.867438 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:12:55.963694 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:55.963654 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:56.942591 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:56.942558 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:12:56.947728 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:12:56.947705 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:13:06.947440 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:06.947403 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:10.286819 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:10.286784 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:13:15.870506 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:15.870467 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:13:22.172884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:22.172846 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:13:24.860067 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:13:24.860027 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:13:26.843134 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:26.843100 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:13:53.256022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:53.255965 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:54.243258 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:54.243221 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:54.859773 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:13:54.859708 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:13:56.080645 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:13:56.080613 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:01.768335 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:01.768295 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:14:03.430501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:03.430463 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:03.806248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:03.806211 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:06.615309 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:06.615265 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:09.761260 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:09.761187 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:14:24.860423 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:14:24.860380 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:14:41.744690 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:41.744653 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:46.345385 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:46.345343 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:14:53.359207 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:53.359169 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:54.350193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:54.350138 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:54.862958 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:14:54.862914 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:14:56.195418 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:14:56.195347 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:05.164240 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:05.164207 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:15:05.546318 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:05.546273 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:05.645725 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:05.645684 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:12.566904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:12.566870 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:15:20.736957 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:20.736908 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:21.138650 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.138386 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6"] Oct 30 04:15:21.141886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.141846 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.156102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.156082 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-26a1d1-openshift-default-dockercfg-q695z\"" Oct 30 04:15:21.169117 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.169086 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6"] Oct 30 04:15:21.177785 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.177736 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:21.261804 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.261748 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-socket\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.261979 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.261884 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-credential-socket\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.261979 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.261931 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b2421-9e73-476f-acf0-afb070276455-istio-podinfo\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.262060 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.262009 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-istio-token\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.262060 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.262044 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-data\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.262128 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.262065 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-envoy\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.262166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.262131 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-certs\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.262202 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.262175 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw9ss\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-kube-api-access-gw9ss\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.262284 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.262266 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b2421-9e73-476f-acf0-afb070276455-istiod-ca-cert\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.315355 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.315327 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:21.363257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363217 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-istio-token\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363275 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-data\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363309 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-envoy\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363336 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-certs\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363376 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw9ss\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-kube-api-access-gw9ss\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363431 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b2421-9e73-476f-acf0-afb070276455-istiod-ca-cert\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363503 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363457 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-socket\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363968 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363523 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-credential-socket\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.363968 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.363576 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b2421-9e73-476f-acf0-afb070276455-istio-podinfo\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.364112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.364086 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-certs\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.364374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.364341 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-data\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.364979 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.364933 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-socket\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.365393 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.365372 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-credential-socket\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.365549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.365443 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b2421-9e73-476f-acf0-afb070276455-istiod-ca-cert\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.368583 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.368561 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-envoy\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.376779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.371251 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b2421-9e73-476f-acf0-afb070276455-istio-podinfo\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.381722 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.381694 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-istio-token\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.382365 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.382344 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw9ss\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-kube-api-access-gw9ss\") pod \"e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.455369 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.455305 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:21.763903 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.763730 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6"] Oct 30 04:15:21.766070 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:15:21.766037 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3b2421_9e73_476f_acf0_afb070276455.slice/crio-d33d6751bfe3d1dc8670bf25263072865d1bbf47b7d81a0f0316ec49e8eccd72 WatchSource:0}: Error finding container d33d6751bfe3d1dc8670bf25263072865d1bbf47b7d81a0f0316ec49e8eccd72: Status 404 returned error can't find the container with id d33d6751bfe3d1dc8670bf25263072865d1bbf47b7d81a0f0316ec49e8eccd72 Oct 30 04:15:21.768141 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.768109 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:15:21.768257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.768191 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:15:21.768257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:21.768240 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:15:22.271553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:22.271515 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" event={"ID":"7d3b2421-9e73-476f-acf0-afb070276455","Type":"ContainerStarted","Data":"4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa"} Oct 30 04:15:22.271732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:22.271566 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" event={"ID":"7d3b2421-9e73-476f-acf0-afb070276455","Type":"ContainerStarted","Data":"d33d6751bfe3d1dc8670bf25263072865d1bbf47b7d81a0f0316ec49e8eccd72"} Oct 30 04:15:22.331194 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:22.331142 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podStartSLOduration=1.331127139 podStartE2EDuration="1.331127139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:22.319604566 +0000 UTC m=+4346.321957766" watchObservedRunningTime="2026-10-30 04:15:22.331127139 +0000 UTC m=+4346.333480336" Oct 30 04:15:22.460846 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:22.456374 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:22.465112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:22.465082 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:23.276415 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:23.276387 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:23.277403 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:23.277385 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:24.708519 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:24.708489 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:24.809863 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:24.809818 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:24.859554 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:15:24.859522 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:15:24.910249 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:24.910215 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:26.118918 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.118882 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs"] Oct 30 04:15:26.121395 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.121374 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.135950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.135928 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-26a1d1-serving-cert\"" Oct 30 04:15:26.149111 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.149084 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs"] Oct 30 04:15:26.196941 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.196910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:26.206950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.206920 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qkp7\" (UniqueName: \"kubernetes.io/projected/59a6aee3-16cf-461f-9001-73f2d38bb4fa-kube-api-access-5qkp7\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.207121 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.206985 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a6aee3-16cf-461f-9001-73f2d38bb4fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.207121 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.207074 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.207214 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.207183 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.297486 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.297451 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:26.308679 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.308646 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.308864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.308709 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qkp7\" (UniqueName: \"kubernetes.io/projected/59a6aee3-16cf-461f-9001-73f2d38bb4fa-kube-api-access-5qkp7\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.308864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.308769 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a6aee3-16cf-461f-9001-73f2d38bb4fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.308864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.308803 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.309517 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.309492 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a6aee3-16cf-461f-9001-73f2d38bb4fa-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.311310 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.311289 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-metrics-certs\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.311422 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.311379 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.333893 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.333808 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qkp7\" (UniqueName: \"kubernetes.io/projected/59a6aee3-16cf-461f-9001-73f2d38bb4fa-kube-api-access-5qkp7\") pod \"maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.402371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.402275 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:26.433280 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.433243 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:26.512398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.512359 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:26.609271 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.609233 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:26.674359 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.674270 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx"] Oct 30 04:15:26.676809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.676793 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:26.689998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.689976 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-26a1d1\"" Oct 30 04:15:26.693809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.693783 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx"] Oct 30 04:15:26.702556 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.702534 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c6c74514-dockercfg-znlwc\"" Oct 30 04:15:26.712729 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.712707 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqlg\" (UniqueName: \"kubernetes.io/projected/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-kube-api-access-9qqlg\") pod \"payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:26.712850 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.712781 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:26.741782 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.741725 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs"] Oct 30 04:15:26.783453 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.782452 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc"] Oct 30 04:15:26.787031 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.787003 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:26.794890 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.794858 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc"] Oct 30 04:15:26.813444 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.813414 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqlg\" (UniqueName: \"kubernetes.io/projected/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-kube-api-access-9qqlg\") pod \"payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:26.813639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.813519 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:26.814428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.814403 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:26.830845 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.830813 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqlg\" (UniqueName: \"kubernetes.io/projected/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-kube-api-access-9qqlg\") pod \"payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:26.914343 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.914309 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xzdv\" (UniqueName: \"kubernetes.io/projected/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-kube-api-access-7xzdv\") pod \"payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:26.914490 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.914358 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:26.989560 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:26.989479 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:27.025342 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.025304 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:27.025531 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.025470 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xzdv\" (UniqueName: \"kubernetes.io/projected/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-kube-api-access-7xzdv\") pod \"payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:27.026537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.026482 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:27.045390 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.045361 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xzdv\" (UniqueName: \"kubernetes.io/projected/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-kube-api-access-7xzdv\") pod \"payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:27.105730 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.105700 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:27.305653 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.305616 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx"] Oct 30 04:15:27.330160 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:15:27.318651 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30cd7d7_9583_4d9b_bdb0_2a724f0642a1.slice/crio-da56162dd9f4c9aa0d6a9b2566a1d34c39954202706a976a854548ed3d4b83d5 WatchSource:0}: Error finding container da56162dd9f4c9aa0d6a9b2566a1d34c39954202706a976a854548ed3d4b83d5: Status 404 returned error can't find the container with id da56162dd9f4c9aa0d6a9b2566a1d34c39954202706a976a854548ed3d4b83d5 Oct 30 04:15:27.330160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.321582 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" event={"ID":"59a6aee3-16cf-461f-9001-73f2d38bb4fa","Type":"ContainerStarted","Data":"b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f"} Oct 30 04:15:27.330160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.321656 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" event={"ID":"59a6aee3-16cf-461f-9001-73f2d38bb4fa","Type":"ContainerStarted","Data":"2f88bcc8e767ffa6b519f6b98b4c2a77af313a19d640218172d2e5766e80276c"} Oct 30 04:15:27.330160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.322018 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:27.366932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.366885 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" podStartSLOduration=0.936950225 podStartE2EDuration="1.366870109s" totalImagesPullingTime="429.919884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:27.365093555 +0000 UTC m=+4351.367446746" watchObservedRunningTime="2026-10-30 04:15:27.366870109 +0000 UTC m=+4351.369223337" Oct 30 04:15:27.400346 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:27.400278 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc"] Oct 30 04:15:27.402506 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:15:27.402477 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3f61a8_e4d6_449a_bce5_40c68320b8d0.slice/crio-92fe7545034a6fa2cb6728e45b6f5ea43c2272586701b198f960064d50c90991 WatchSource:0}: Error finding container 92fe7545034a6fa2cb6728e45b6f5ea43c2272586701b198f960064d50c90991: Status 404 returned error can't find the container with id 92fe7545034a6fa2cb6728e45b6f5ea43c2272586701b198f960064d50c90991 Oct 30 04:15:28.010308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.010274 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:28.327166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.327071 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" event={"ID":"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1","Type":"ContainerStarted","Data":"1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9"} Oct 30 04:15:28.327166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.327140 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" event={"ID":"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1","Type":"ContainerStarted","Data":"da56162dd9f4c9aa0d6a9b2566a1d34c39954202706a976a854548ed3d4b83d5"} Oct 30 04:15:28.327692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.327169 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:28.328666 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.328637 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" event={"ID":"3e3f61a8-e4d6-449a-bce5-40c68320b8d0","Type":"ContainerStarted","Data":"328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3"} Oct 30 04:15:28.328831 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.328682 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" event={"ID":"3e3f61a8-e4d6-449a-bce5-40c68320b8d0","Type":"ContainerStarted","Data":"92fe7545034a6fa2cb6728e45b6f5ea43c2272586701b198f960064d50c90991"} Oct 30 04:15:28.328941 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.328928 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:28.364541 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.364494 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" podStartSLOduration=2.364482996 podStartE2EDuration="2.364482996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:28.362190195 +0000 UTC m=+4352.364543392" watchObservedRunningTime="2026-10-30 04:15:28.364482996 +0000 UTC m=+4352.366836193" Oct 30 04:15:28.399416 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:28.399359 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" podStartSLOduration=2.39934357 podStartE2EDuration="2.39934357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:28.396951152 +0000 UTC m=+4352.399304353" watchObservedRunningTime="2026-10-30 04:15:28.39934357 +0000 UTC m=+4352.401696768" Oct 30 04:15:29.712332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:29.712293 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:29.939163 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:29.939131 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:15:31.506611 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:31.506563 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:33.334414 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:33.334385 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:15:33.337930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:33.337907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:35.078736 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:35.078702 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:36.925442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:36.925351 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:38.535412 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:38.535378 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:39.316953 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:39.316868 4442 patch_prober.go:29] interesting pod/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:15:39.317115 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:39.316944 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:15:39.335990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:39.335967 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:15:39.336128 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:39.336029 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:15:39.639238 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:39.638575 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:40.281467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.281435 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:40.389782 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.385582 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:40.394552 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.392029 4442 generic.go:356] "Generic (PLEG): container finished" podID="7d3b2421-9e73-476f-acf0-afb070276455" containerID="4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa" exitCode=0 Oct 30 04:15:40.394552 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.392164 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" event={"ID":"7d3b2421-9e73-476f-acf0-afb070276455","Type":"ContainerDied","Data":"4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa"} Oct 30 04:15:40.394552 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.392577 4442 scope.go:122] "RemoveContainer" containerID="4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa" Oct 30 04:15:40.395791 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.395733 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:15:40.395938 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.395818 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:15:40.395938 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.395864 4442 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30891996Ki","pods":"250"} Oct 30 04:15:40.480461 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:40.480416 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:41.412528 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:41.412491 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" event={"ID":"7d3b2421-9e73-476f-acf0-afb070276455","Type":"ContainerStarted","Data":"1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d"} Oct 30 04:15:41.455668 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:41.455617 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:41.455947 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:41.455923 4442 patch_prober.go:29] interesting pod/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 04:15:41.456007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:41.455975 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 04:15:42.105264 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:42.105228 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:42.353770 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:42.353724 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:42.456104 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:42.456025 4442 patch_prober.go:29] interesting pod/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 04:15:42.456104 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:42.456084 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 04:15:43.456193 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:43.456159 4442 patch_prober.go:29] interesting pod/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 04:15:43.456620 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:43.456212 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 04:15:43.828187 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:43.828155 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:44.455892 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:44.455859 4442 patch_prober.go:29] interesting pod/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 04:15:44.456111 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:44.455925 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 04:15:44.673864 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:44.673826 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:15:45.456624 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:45.456590 4442 patch_prober.go:29] interesting pod/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" start-of-body= Oct 30 04:15:45.456843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:45.456649 4442 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.251:15021/healthz/ready\": dial tcp 10.134.0.251:15021: connect: connection refused" Oct 30 04:15:45.561720 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:45.561670 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:46.849514 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:46.849481 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:46.850109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:46.850060 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:46.851048 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:46.851024 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:15:47.231368 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:47.231277 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:48.903609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:48.903573 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:50.614944 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:50.614910 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:52.310936 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:52.310905 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:53.477929 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:53.477896 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:54.029059 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:54.029001 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:54.456224 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:54.456130 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:54.860250 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:15:54.860208 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:15:55.724865 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:55.724833 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:56.314656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:56.314624 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:57.410141 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:57.410107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:59.108109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:15:59.108076 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:00.134737 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:00.134701 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:16:00.913413 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:00.913383 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:02.603395 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:02.603359 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:04.089944 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:04.089909 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:05.592123 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:05.592079 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:06.018334 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:06.018307 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:06.115706 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:06.115658 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:07.116876 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:07.116843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:07.217710 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:07.217675 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:08.531951 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:08.531921 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:09.972434 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:09.972400 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:11.399639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:11.399598 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:12.896134 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:12.896092 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:14.495587 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:14.495555 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:16.039507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:16.039473 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:17.603372 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:17.603335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:19.202927 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:19.202895 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:20.706793 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:20.706742 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:22.290874 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:22.290836 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:23.053466 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:23.053432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:23.521166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:23.521136 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:23.670220 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:23.670186 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:16:23.797781 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:23.797688 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:24.860516 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:16:24.860470 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:16:25.298054 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:25.298024 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:26.805156 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:26.805124 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:28.097810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:28.097774 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:28.299114 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:28.299083 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:29.388742 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:29.388706 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc"] Oct 30 04:16:29.423813 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:29.423780 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx"] Oct 30 04:16:29.527278 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:29.527246 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:29.627381 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:29.627349 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:29.920744 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:29.920714 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:30.646745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:30.646713 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:31.268237 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:31.268207 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs"] Oct 30 04:16:31.268423 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:31.268390 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" podUID="59a6aee3-16cf-461f-9001-73f2d38bb4fa" containerName="maas-api" containerID="cri-o://b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f" gracePeriod=30 Oct 30 04:16:31.328646 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:31.328615 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:32.522458 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.522436 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:16:32.641727 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.641635 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-maas-api-tls\") pod \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " Oct 30 04:16:32.641727 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.641692 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-metrics-certs\") pod \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " Oct 30 04:16:32.642029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.641860 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a6aee3-16cf-461f-9001-73f2d38bb4fa-kube-api-access-5qkp7\" (UniqueName: \"kubernetes.io/projected/59a6aee3-16cf-461f-9001-73f2d38bb4fa-kube-api-access-5qkp7\") pod \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " Oct 30 04:16:32.642029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.641906 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a6aee3-16cf-461f-9001-73f2d38bb4fa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a6aee3-16cf-461f-9001-73f2d38bb4fa-openshift-service-ca\") pod \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\" (UID: \"59a6aee3-16cf-461f-9001-73f2d38bb4fa\") " Oct 30 04:16:32.642341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.642314 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a6aee3-16cf-461f-9001-73f2d38bb4fa-openshift-service-ca" pod "59a6aee3-16cf-461f-9001-73f2d38bb4fa" (UID: "59a6aee3-16cf-461f-9001-73f2d38bb4fa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:32.643917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.643891 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-maas-api-tls" pod "59a6aee3-16cf-461f-9001-73f2d38bb4fa" (UID: "59a6aee3-16cf-461f-9001-73f2d38bb4fa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:32.644030 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.643986 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a6aee3-16cf-461f-9001-73f2d38bb4fa-kube-api-access-5qkp7" pod "59a6aee3-16cf-461f-9001-73f2d38bb4fa" (UID: "59a6aee3-16cf-461f-9001-73f2d38bb4fa"). InnerVolumeSpecName "kube-api-access-5qkp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:32.644093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.644026 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-metrics-certs" pod "59a6aee3-16cf-461f-9001-73f2d38bb4fa" (UID: "59a6aee3-16cf-461f-9001-73f2d38bb4fa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:16:32.681010 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.680980 4442 generic.go:356] "Generic (PLEG): container finished" podID="59a6aee3-16cf-461f-9001-73f2d38bb4fa" containerID="b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f" exitCode=0 Oct 30 04:16:32.681115 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.681031 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" Oct 30 04:16:32.681115 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.681063 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" event={"ID":"59a6aee3-16cf-461f-9001-73f2d38bb4fa","Type":"ContainerDied","Data":"b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f"} Oct 30 04:16:32.681197 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.681116 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs" event={"ID":"59a6aee3-16cf-461f-9001-73f2d38bb4fa","Type":"ContainerDied","Data":"2f88bcc8e767ffa6b519f6b98b4c2a77af313a19d640218172d2e5766e80276c"} Oct 30 04:16:32.681197 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.681138 4442 scope.go:122] "RemoveContainer" containerID="b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f" Oct 30 04:16:32.690357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.690341 4442 scope.go:122] "RemoveContainer" containerID="b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f" Oct 30 04:16:32.690626 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:16:32.690604 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f\": container with ID starting with b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f not found: ID does not exist" containerID="b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f" Oct 30 04:16:32.690696 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.690637 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f"} err="failed to get container status \"b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f\": rpc error: code = NotFound desc = could not find container \"b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f\": container with ID starting with b7e18411f61cd263f02d6d7f00df4a5973953e850cfd18043965fbc35314a06f not found: ID does not exist" Oct 30 04:16:32.727394 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.727350 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs"] Oct 30 04:16:32.729052 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.729033 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-26a1d1-678b8f6dfb-qmjrs"] Oct 30 04:16:32.742963 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.742939 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qkp7\" (UniqueName: \"kubernetes.io/projected/59a6aee3-16cf-461f-9001-73f2d38bb4fa-kube-api-access-5qkp7\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:32.742963 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.742960 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59a6aee3-16cf-461f-9001-73f2d38bb4fa-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:32.743091 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.742970 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:32.743091 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:32.742979 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59a6aee3-16cf-461f-9001-73f2d38bb4fa-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:33.725550 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:33.725520 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:16:34.105222 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:34.105191 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a6aee3-16cf-461f-9001-73f2d38bb4fa" path="/var/lib/kubelet/pods/59a6aee3-16cf-461f-9001-73f2d38bb4fa/volumes" Oct 30 04:16:34.389305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:34.389216 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" podUID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" containerName="payload-processing" containerID="cri-o://328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3" gracePeriod=25 Oct 30 04:16:34.424799 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:34.424721 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" podUID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" containerName="payload-pre-processing" containerID="cri-o://1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9" gracePeriod=25 Oct 30 04:16:36.549480 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:36.549448 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:36.649534 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:36.649486 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:37.244301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:37.244263 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:37.626862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:37.626118 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6"] Oct 30 04:16:37.626862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:37.626409 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" containerID="cri-o://1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d" gracePeriod=30 Oct 30 04:16:37.671635 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:37.671598 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:39.334371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:39.334327 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" podUID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:16:39.334371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:39.334352 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" podUID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:16:42.141339 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.141279 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:42.882208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.882184 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:16:42.945283 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945203 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-istio-token\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945283 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945267 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-envoy\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945295 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-credential-socket\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945396 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-socket\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945441 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-kube-api-access-gw9ss\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-kube-api-access-gw9ss\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945513 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945471 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d3b2421-9e73-476f-acf0-afb070276455-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b2421-9e73-476f-acf0-afb070276455-istiod-ca-cert\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945784 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945528 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-certs\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945784 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945566 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d3b2421-9e73-476f-acf0-afb070276455-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b2421-9e73-476f-acf0-afb070276455-istio-podinfo\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945784 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945617 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-data\") pod \"7d3b2421-9e73-476f-acf0-afb070276455\" (UID: \"7d3b2421-9e73-476f-acf0-afb070276455\") " Oct 30 04:16:42.945984 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.945957 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-credential-socket" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:42.946198 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.946171 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-certs" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:42.946570 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.946542 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-socket" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:42.946570 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.946560 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3b2421-9e73-476f-acf0-afb070276455-istiod-ca-cert" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:42.946771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.946735 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-data" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:42.947921 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.947890 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-envoy" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:16:42.948248 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.948221 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-kube-api-access-gw9ss" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "kube-api-access-gw9ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:42.948532 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.948508 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-istio-token" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:42.948917 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:42.948899 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d3b2421-9e73-476f-acf0-afb070276455-istio-podinfo" pod "7d3b2421-9e73-476f-acf0-afb070276455" (UID: "7d3b2421-9e73-476f-acf0-afb070276455"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:16:43.046889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046860 4442 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.046889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046885 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw9ss\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-kube-api-access-gw9ss\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.046889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046896 4442 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d3b2421-9e73-476f-acf0-afb070276455-istiod-ca-cert\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.046889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046906 4442 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-workload-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.047161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046915 4442 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d3b2421-9e73-476f-acf0-afb070276455-istio-podinfo\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.047161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046923 4442 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-data\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.047161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046932 4442 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d3b2421-9e73-476f-acf0-afb070276455-istio-token\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.047161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046940 4442 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-istio-envoy\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.047161 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.046959 4442 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d3b2421-9e73-476f-acf0-afb070276455-credential-socket\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:43.740301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.740268 4442 generic.go:356] "Generic (PLEG): container finished" podID="7d3b2421-9e73-476f-acf0-afb070276455" containerID="1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d" exitCode=0 Oct 30 04:16:43.740884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.740313 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" event={"ID":"7d3b2421-9e73-476f-acf0-afb070276455","Type":"ContainerDied","Data":"1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d"} Oct 30 04:16:43.740884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.740334 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" Oct 30 04:16:43.740884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.740355 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6" event={"ID":"7d3b2421-9e73-476f-acf0-afb070276455","Type":"ContainerDied","Data":"d33d6751bfe3d1dc8670bf25263072865d1bbf47b7d81a0f0316ec49e8eccd72"} Oct 30 04:16:43.740884 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.740376 4442 scope.go:122] "RemoveContainer" containerID="1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d" Oct 30 04:16:43.749891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.749644 4442 scope.go:122] "RemoveContainer" containerID="4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa" Oct 30 04:16:43.757446 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.757426 4442 scope.go:122] "RemoveContainer" containerID="1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d" Oct 30 04:16:43.757707 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:16:43.757690 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d\": container with ID starting with 1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d not found: ID does not exist" containerID="1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d" Oct 30 04:16:43.757769 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.757717 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d"} err="failed to get container status \"1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d\": rpc error: code = NotFound desc = could not find container \"1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d\": container with ID starting with 1045e47fad28ec255347a7397e0ff91cd2fc29ef29b12844a487165cfaee775d not found: ID does not exist" Oct 30 04:16:43.757769 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.757736 4442 scope.go:122] "RemoveContainer" containerID="4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa" Oct 30 04:16:43.758014 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:16:43.758000 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa\": container with ID starting with 4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa not found: ID does not exist" containerID="4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa" Oct 30 04:16:43.758055 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.758018 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa"} err="failed to get container status \"4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa\": rpc error: code = NotFound desc = could not find container \"4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa\": container with ID starting with 4e7f1f14774d99950c3f77c88cf5fe4a3baf34c9b75eb18529936f3262014aaa not found: ID does not exist" Oct 30 04:16:43.801732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.801707 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6"] Oct 30 04:16:43.812711 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:43.812691 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-26a1d1-openshift-default-5d4454dcb5dhkx6"] Oct 30 04:16:44.106406 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:44.106362 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d3b2421-9e73-476f-acf0-afb070276455" path="/var/lib/kubelet/pods/7d3b2421-9e73-476f-acf0-afb070276455/volumes" Oct 30 04:16:44.557898 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:44.557867 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:16:49.334804 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:49.334746 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" podUID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:16:49.334804 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:49.334778 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" podUID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:16:53.600287 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:53.600248 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:54.564045 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.563970 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:54.672599 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.672574 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:16:54.699224 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.699202 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:16:54.789458 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.789427 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-plugins-config-volume\") pod \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " Oct 30 04:16:54.789644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.789516 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-kube-api-access-9qqlg\" (UniqueName: \"kubernetes.io/projected/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-kube-api-access-9qqlg\") pod \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " Oct 30 04:16:54.789644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.789579 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-kube-api-access-7xzdv\" (UniqueName: \"kubernetes.io/projected/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-kube-api-access-7xzdv\") pod \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\" (UID: \"3e3f61a8-e4d6-449a-bce5-40c68320b8d0\") " Oct 30 04:16:54.789644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.789612 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-plugins-config-volume\") pod \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\" (UID: \"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1\") " Oct 30 04:16:54.789895 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.789874 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-plugins-config-volume" pod "3e3f61a8-e4d6-449a-bce5-40c68320b8d0" (UID: "3e3f61a8-e4d6-449a-bce5-40c68320b8d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:54.790139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.790107 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-plugins-config-volume" pod "b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" (UID: "b30cd7d7-9583-4d9b-bdb0-2a724f0642a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:16:54.790139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.790117 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:54.791904 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.791884 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-kube-api-access-7xzdv" pod "3e3f61a8-e4d6-449a-bce5-40c68320b8d0" (UID: "3e3f61a8-e4d6-449a-bce5-40c68320b8d0"). InnerVolumeSpecName "kube-api-access-7xzdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:54.791969 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.791913 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-kube-api-access-9qqlg" pod "b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" (UID: "b30cd7d7-9583-4d9b-bdb0-2a724f0642a1"). InnerVolumeSpecName "kube-api-access-9qqlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:16:54.809783 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.809736 4442 generic.go:356] "Generic (PLEG): container finished" podID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" containerID="328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3" exitCode=0 Oct 30 04:16:54.809925 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.809805 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" Oct 30 04:16:54.809925 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.809825 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" event={"ID":"3e3f61a8-e4d6-449a-bce5-40c68320b8d0","Type":"ContainerDied","Data":"328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3"} Oct 30 04:16:54.810053 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.809922 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc" event={"ID":"3e3f61a8-e4d6-449a-bce5-40c68320b8d0","Type":"ContainerDied","Data":"92fe7545034a6fa2cb6728e45b6f5ea43c2272586701b198f960064d50c90991"} Oct 30 04:16:54.810053 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.809958 4442 scope.go:122] "RemoveContainer" containerID="328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3" Oct 30 04:16:54.811366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.811343 4442 generic.go:356] "Generic (PLEG): container finished" podID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" containerID="1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9" exitCode=0 Oct 30 04:16:54.811628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.811376 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" event={"ID":"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1","Type":"ContainerDied","Data":"1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9"} Oct 30 04:16:54.811628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.811404 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" Oct 30 04:16:54.811628 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.811444 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx" event={"ID":"b30cd7d7-9583-4d9b-bdb0-2a724f0642a1","Type":"ContainerDied","Data":"da56162dd9f4c9aa0d6a9b2566a1d34c39954202706a976a854548ed3d4b83d5"} Oct 30 04:16:54.818512 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.818494 4442 scope.go:122] "RemoveContainer" containerID="328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3" Oct 30 04:16:54.818788 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:16:54.818765 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3\": container with ID starting with 328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3 not found: ID does not exist" containerID="328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3" Oct 30 04:16:54.818841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.818802 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3"} err="failed to get container status \"328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3\": rpc error: code = NotFound desc = could not find container \"328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3\": container with ID starting with 328fb739c802b05accbce11e61b59492bf8b84a64aa0bf0848759dff6ee965e3 not found: ID does not exist" Oct 30 04:16:54.818841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.818824 4442 scope.go:122] "RemoveContainer" containerID="1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9" Oct 30 04:16:54.827360 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.827344 4442 scope.go:122] "RemoveContainer" containerID="1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9" Oct 30 04:16:54.827620 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:16:54.827599 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9\": container with ID starting with 1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9 not found: ID does not exist" containerID="1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9" Oct 30 04:16:54.827669 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.827628 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9"} err="failed to get container status \"1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9\": rpc error: code = NotFound desc = could not find container \"1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9\": container with ID starting with 1dd7fad4fc00a496a90df0ccbfcdacce0a86cde7bfe239ada530bc8d1fe515c9 not found: ID does not exist" Oct 30 04:16:54.856196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.856164 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx"] Oct 30 04:16:54.860330 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.860302 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-26a1d1-5bc8b8srfx"] Oct 30 04:16:54.861034 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:16:54.861003 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:16:54.890081 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.890059 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc"] Oct 30 04:16:54.890910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.890889 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:54.890970 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.890915 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qqlg\" (UniqueName: \"kubernetes.io/projected/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1-kube-api-access-9qqlg\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:54.890970 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.890927 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xzdv\" (UniqueName: \"kubernetes.io/projected/3e3f61a8-e4d6-449a-bce5-40c68320b8d0-kube-api-access-7xzdv\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:16:54.893844 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:54.893821 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-26a1d1-5979c8ddfbdcgc"] Oct 30 04:16:56.105471 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:56.105441 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" path="/var/lib/kubelet/pods/3e3f61a8-e4d6-449a-bce5-40c68320b8d0/volumes" Oct 30 04:16:56.105885 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:56.105832 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" path="/var/lib/kubelet/pods/b30cd7d7-9583-4d9b-bdb0-2a724f0642a1/volumes" Oct 30 04:16:56.416967 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:56.416934 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:58.301348 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301316 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m"] Oct 30 04:16:58.301838 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301818 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" Oct 30 04:16:58.301932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301841 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" Oct 30 04:16:58.301932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301888 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" containerName="payload-pre-processing" Oct 30 04:16:58.301932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301897 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" containerName="payload-pre-processing" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301935 4442 cpu_manager.go:414] "containerMap: removing container" podUID="59a6aee3-16cf-461f-9001-73f2d38bb4fa" containerName="maas-api" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301941 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a6aee3-16cf-461f-9001-73f2d38bb4fa" containerName="maas-api" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301953 4442 cpu_manager.go:414] "containerMap: removing container" podUID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" containerName="payload-processing" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.301959 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" containerName="payload-processing" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.302049 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.302064 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30cd7d7-9583-4d9b-bdb0-2a724f0642a1" containerName="payload-pre-processing" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.302076 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a6aee3-16cf-461f-9001-73f2d38bb4fa" containerName="maas-api" Oct 30 04:16:58.302101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.302081 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e3f61a8-e4d6-449a-bce5-40c68320b8d0" containerName="payload-processing" Oct 30 04:16:58.306644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.306616 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.323306 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.323231 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m"] Oct 30 04:16:58.332246 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.332225 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d820fa78-kserve-self-signed-certs\"" Oct 30 04:16:58.353157 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.353127 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:58.424291 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.424261 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-model-cache\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.424431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.424297 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9kw\" (UniqueName: \"kubernetes.io/projected/23b9583c-a12f-414a-a565-92ee9f06ec2c-kube-api-access-xd9kw\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.424431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.424366 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-dshm\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.424431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.424410 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-home\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.424431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.424428 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9583c-a12f-414a-a565-92ee9f06ec2c-tls-certs\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.424571 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.424476 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-tmp-dir\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.525490 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.525448 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-model-cache\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.525719 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.525508 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9kw\" (UniqueName: \"kubernetes.io/projected/23b9583c-a12f-414a-a565-92ee9f06ec2c-kube-api-access-xd9kw\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.525719 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.525578 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-dshm\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.525719 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.525615 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-home\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.525719 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.525638 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9583c-a12f-414a-a565-92ee9f06ec2c-tls-certs\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.525719 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.525658 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-tmp-dir\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.526075 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.526053 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-tmp-dir\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.526142 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.526075 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-model-cache\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.526428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.526321 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-home\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.532789 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.530407 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-dshm\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.532934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.532851 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9583c-a12f-414a-a565-92ee9f06ec2c-tls-certs\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.543510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.543489 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9kw\" (UniqueName: \"kubernetes.io/projected/23b9583c-a12f-414a-a565-92ee9f06ec2c-kube-api-access-xd9kw\") pod \"e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.618997 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.618920 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:16:58.912191 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:58.912167 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m"] Oct 30 04:16:58.914679 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:16:58.914655 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23b9583c_a12f_414a_a565_92ee9f06ec2c.slice/crio-4234690f7fac7673c0358b44bbb7c2b94e4f1387c27f45f1d21e0873df9f5904 WatchSource:0}: Error finding container 4234690f7fac7673c0358b44bbb7c2b94e4f1387c27f45f1d21e0873df9f5904: Status 404 returned error can't find the container with id 4234690f7fac7673c0358b44bbb7c2b94e4f1387c27f45f1d21e0873df9f5904 Oct 30 04:16:59.837257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:59.837217 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" event={"ID":"23b9583c-a12f-414a-a565-92ee9f06ec2c","Type":"ContainerStarted","Data":"6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04"} Oct 30 04:16:59.837679 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:59.837265 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" event={"ID":"23b9583c-a12f-414a-a565-92ee9f06ec2c","Type":"ContainerStarted","Data":"4234690f7fac7673c0358b44bbb7c2b94e4f1387c27f45f1d21e0873df9f5904"} Oct 30 04:16:59.874385 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:16:59.874336 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" podStartSLOduration=1.8743218339999999 podStartE2EDuration="1.874321834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:59.873794297 +0000 UTC m=+4443.876147496" watchObservedRunningTime="2026-10-30 04:16:59.874321834 +0000 UTC m=+4443.876675031" Oct 30 04:17:00.619795 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:00.619744 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:17:00.666270 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:00.666239 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:17:05.240301 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:05.240267 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:05.386870 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:05.386839 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:08.619118 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:08.619085 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:17:08.631712 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:08.631687 4442 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:17:08.896004 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:08.895919 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:17:10.925930 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:10.925900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:17.039196 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:17.039162 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:17:24.860220 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:17:24.860184 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:17:26.752371 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:26.752335 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:28.170647 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:28.170615 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m"] Oct 30 04:17:28.197766 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:28.197730 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:28.930742 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:28.930710 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:17:36.057129 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:36.057099 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:17:43.191955 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.191902 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" podUID="23b9583c-a12f-414a-a565-92ee9f06ec2c" containerName="main" containerID="cri-o://6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04" gracePeriod=45 Oct 30 04:17:43.445034 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.444976 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:17:43.604553 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.604523 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b9583c-a12f-414a-a565-92ee9f06ec2c-tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9583c-a12f-414a-a565-92ee9f06ec2c-tls-certs\") pod \"23b9583c-a12f-414a-a565-92ee9f06ec2c\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " Oct 30 04:17:43.604732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.604586 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-model-cache\") pod \"23b9583c-a12f-414a-a565-92ee9f06ec2c\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " Oct 30 04:17:43.604732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.604666 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-home\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-home\") pod \"23b9583c-a12f-414a-a565-92ee9f06ec2c\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " Oct 30 04:17:43.604732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.604710 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-dshm\") pod \"23b9583c-a12f-414a-a565-92ee9f06ec2c\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " Oct 30 04:17:43.604732 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.604728 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-tmp-dir\") pod \"23b9583c-a12f-414a-a565-92ee9f06ec2c\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " Oct 30 04:17:43.604999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.604789 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b9583c-a12f-414a-a565-92ee9f06ec2c-kube-api-access-xd9kw\" (UniqueName: \"kubernetes.io/projected/23b9583c-a12f-414a-a565-92ee9f06ec2c-kube-api-access-xd9kw\") pod \"23b9583c-a12f-414a-a565-92ee9f06ec2c\" (UID: \"23b9583c-a12f-414a-a565-92ee9f06ec2c\") " Oct 30 04:17:43.604999 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.604922 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-model-cache" pod "23b9583c-a12f-414a-a565-92ee9f06ec2c" (UID: "23b9583c-a12f-414a-a565-92ee9f06ec2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:43.605130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.605031 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-home" pod "23b9583c-a12f-414a-a565-92ee9f06ec2c" (UID: "23b9583c-a12f-414a-a565-92ee9f06ec2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:43.605130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.605054 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-tmp-dir" pod "23b9583c-a12f-414a-a565-92ee9f06ec2c" (UID: "23b9583c-a12f-414a-a565-92ee9f06ec2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:43.605215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.605187 4442 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-home\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:17:43.605215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.605207 4442 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-tmp-dir\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:17:43.605292 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.605221 4442 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-model-cache\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:17:43.607093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.607069 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-dshm" pod "23b9583c-a12f-414a-a565-92ee9f06ec2c" (UID: "23b9583c-a12f-414a-a565-92ee9f06ec2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:43.607093 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.607080 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b9583c-a12f-414a-a565-92ee9f06ec2c-kube-api-access-xd9kw" pod "23b9583c-a12f-414a-a565-92ee9f06ec2c" (UID: "23b9583c-a12f-414a-a565-92ee9f06ec2c"). InnerVolumeSpecName "kube-api-access-xd9kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:43.607236 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.607075 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b9583c-a12f-414a-a565-92ee9f06ec2c-tls-certs" pod "23b9583c-a12f-414a-a565-92ee9f06ec2c" (UID: "23b9583c-a12f-414a-a565-92ee9f06ec2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:43.705961 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.705881 4442 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23b9583c-a12f-414a-a565-92ee9f06ec2c-dshm\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:17:43.705961 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.705911 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd9kw\" (UniqueName: \"kubernetes.io/projected/23b9583c-a12f-414a-a565-92ee9f06ec2c-kube-api-access-xd9kw\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:17:43.705961 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.705923 4442 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23b9583c-a12f-414a-a565-92ee9f06ec2c-tls-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:17:43.960283 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:43.960187 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:17:44.275333 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.275297 4442 generic.go:356] "Generic (PLEG): container finished" podID="23b9583c-a12f-414a-a565-92ee9f06ec2c" containerID="6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04" exitCode=0 Oct 30 04:17:44.275779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.275364 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" Oct 30 04:17:44.275779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.275382 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" event={"ID":"23b9583c-a12f-414a-a565-92ee9f06ec2c","Type":"ContainerDied","Data":"6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04"} Oct 30 04:17:44.275779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.275432 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m" event={"ID":"23b9583c-a12f-414a-a565-92ee9f06ec2c","Type":"ContainerDied","Data":"4234690f7fac7673c0358b44bbb7c2b94e4f1387c27f45f1d21e0873df9f5904"} Oct 30 04:17:44.275779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.275449 4442 scope.go:122] "RemoveContainer" containerID="6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04" Oct 30 04:17:44.283891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.283873 4442 scope.go:122] "RemoveContainer" containerID="6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04" Oct 30 04:17:44.284161 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:17:44.284139 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04\": container with ID starting with 6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04 not found: ID does not exist" containerID="6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04" Oct 30 04:17:44.284212 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.284169 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04"} err="failed to get container status \"6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04\": rpc error: code = NotFound desc = could not find container \"6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04\": container with ID starting with 6718c4775585db54871236ac6ddce592d0c86d17d9b1932153f71ab302ebeb04 not found: ID does not exist" Oct 30 04:17:44.322243 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.322212 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m"] Oct 30 04:17:44.334467 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:44.334428 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d820fa78-kserve-66b466f7c-95l4m"] Oct 30 04:17:46.137778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:46.137707 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b9583c-a12f-414a-a565-92ee9f06ec2c" path="/var/lib/kubelet/pods/23b9583c-a12f-414a-a565-92ee9f06ec2c/volumes" Oct 30 04:17:51.383091 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:51.383052 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:17:53.712160 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:53.712123 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:54.787319 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:54.787282 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:54.860871 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:17:54.860835 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:17:56.530336 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:56.530299 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:57.020144 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:57.020120 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:17:57.042515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:17:57.042483 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:18:02.580102 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:02.580063 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:18:18.967479 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:18.967445 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v"] Oct 30 04:18:23.967829 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:23.967782 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" podUID="ca299bea-7668-4ecc-80e9-73dcefad570d" containerName="payload-pre-processing" containerID="cri-o://f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817" gracePeriod=25 Oct 30 04:18:24.224072 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.224013 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 04:18:24.329100 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.329073 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca299bea-7668-4ecc-80e9-73dcefad570d-kube-api-access-m8hgj\" (UniqueName: \"kubernetes.io/projected/ca299bea-7668-4ecc-80e9-73dcefad570d-kube-api-access-m8hgj\") pod \"ca299bea-7668-4ecc-80e9-73dcefad570d\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " Oct 30 04:18:24.329306 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.329120 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca299bea-7668-4ecc-80e9-73dcefad570d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca299bea-7668-4ecc-80e9-73dcefad570d-plugins-config-volume\") pod \"ca299bea-7668-4ecc-80e9-73dcefad570d\" (UID: \"ca299bea-7668-4ecc-80e9-73dcefad570d\") " Oct 30 04:18:24.329538 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.329513 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca299bea-7668-4ecc-80e9-73dcefad570d-plugins-config-volume" pod "ca299bea-7668-4ecc-80e9-73dcefad570d" (UID: "ca299bea-7668-4ecc-80e9-73dcefad570d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:24.331357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.331331 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca299bea-7668-4ecc-80e9-73dcefad570d-kube-api-access-m8hgj" pod "ca299bea-7668-4ecc-80e9-73dcefad570d" (UID: "ca299bea-7668-4ecc-80e9-73dcefad570d"). InnerVolumeSpecName "kube-api-access-m8hgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:24.430003 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.429959 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hgj\" (UniqueName: \"kubernetes.io/projected/ca299bea-7668-4ecc-80e9-73dcefad570d-kube-api-access-m8hgj\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:18:24.430003 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.430002 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca299bea-7668-4ecc-80e9-73dcefad570d-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:18:24.524262 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.524232 4442 generic.go:356] "Generic (PLEG): container finished" podID="ca299bea-7668-4ecc-80e9-73dcefad570d" containerID="f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817" exitCode=0 Oct 30 04:18:24.524421 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.524284 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" Oct 30 04:18:24.524421 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.524287 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" event={"ID":"ca299bea-7668-4ecc-80e9-73dcefad570d","Type":"ContainerDied","Data":"f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817"} Oct 30 04:18:24.524421 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.524414 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v" event={"ID":"ca299bea-7668-4ecc-80e9-73dcefad570d","Type":"ContainerDied","Data":"1b86f95b12ccced3f1cff23333d016827754f81f00d832a4568c6e9a11e2ff7a"} Oct 30 04:18:24.524576 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.524447 4442 scope.go:122] "RemoveContainer" containerID="f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817" Oct 30 04:18:24.533085 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.533067 4442 scope.go:122] "RemoveContainer" containerID="f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817" Oct 30 04:18:24.533353 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:18:24.533334 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817\": container with ID starting with f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817 not found: ID does not exist" containerID="f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817" Oct 30 04:18:24.533424 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.533362 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817"} err="failed to get container status \"f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817\": rpc error: code = NotFound desc = could not find container \"f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817\": container with ID starting with f729d9e0bb85d6137b0f9c6ff85091b1a939a93ca2b9eac33e6545e46d411817 not found: ID does not exist" Oct 30 04:18:24.570192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.570161 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v"] Oct 30 04:18:24.580807 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:24.580784 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-n9x2v"] Oct 30 04:18:24.860225 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:18:24.860133 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:18:26.104815 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:26.104784 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca299bea-7668-4ecc-80e9-73dcefad570d" path="/var/lib/kubelet/pods/ca299bea-7668-4ecc-80e9-73dcefad570d/volumes" Oct 30 04:18:33.058077 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:33.058046 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hgcd9"] Oct 30 04:18:35.019822 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.019783 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58ddc49fd8-p6cp9"] Oct 30 04:18:35.020261 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020245 4442 cpu_manager.go:414] "containerMap: removing container" podUID="ca299bea-7668-4ecc-80e9-73dcefad570d" containerName="payload-pre-processing" Oct 30 04:18:35.020308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020263 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca299bea-7668-4ecc-80e9-73dcefad570d" containerName="payload-pre-processing" Oct 30 04:18:35.020308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020275 4442 cpu_manager.go:414] "containerMap: removing container" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" Oct 30 04:18:35.020308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020281 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" Oct 30 04:18:35.020308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020304 4442 cpu_manager.go:414] "containerMap: removing container" podUID="23b9583c-a12f-414a-a565-92ee9f06ec2c" containerName="main" Oct 30 04:18:35.020308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020309 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b9583c-a12f-414a-a565-92ee9f06ec2c" containerName="main" Oct 30 04:18:35.020478 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020364 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca299bea-7668-4ecc-80e9-73dcefad570d" containerName="payload-pre-processing" Oct 30 04:18:35.020478 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020374 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d3b2421-9e73-476f-acf0-afb070276455" containerName="istio-proxy" Oct 30 04:18:35.020478 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.020383 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b9583c-a12f-414a-a565-92ee9f06ec2c" containerName="main" Oct 30 04:18:35.024896 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.024874 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.041708 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.041684 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58ddc49fd8-p6cp9"] Oct 30 04:18:35.120339 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.120306 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9921584-7832-4cc4-acde-3750a3d8f196-plugins-config-volume\") pod \"payload-processing-58ddc49fd8-p6cp9\" (UID: \"b9921584-7832-4cc4-acde-3750a3d8f196\") " pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.120528 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.120387 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7fkx\" (UniqueName: \"kubernetes.io/projected/b9921584-7832-4cc4-acde-3750a3d8f196-kube-api-access-n7fkx\") pod \"payload-processing-58ddc49fd8-p6cp9\" (UID: \"b9921584-7832-4cc4-acde-3750a3d8f196\") " pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.196487 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.196454 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-cf6c69f79-89fc9"] Oct 30 04:18:35.199879 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.199860 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.217549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.217518 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-cf6c69f79-89fc9"] Oct 30 04:18:35.221114 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.221092 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9921584-7832-4cc4-acde-3750a3d8f196-plugins-config-volume\") pod \"payload-processing-58ddc49fd8-p6cp9\" (UID: \"b9921584-7832-4cc4-acde-3750a3d8f196\") " pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.221232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.221132 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndv2\" (UniqueName: \"kubernetes.io/projected/ecd2a582-cbbc-4f33-9214-ffbd548bff75-kube-api-access-mndv2\") pod \"payload-pre-processing-cf6c69f79-89fc9\" (UID: \"ecd2a582-cbbc-4f33-9214-ffbd548bff75\") " pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.221232 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.221172 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7fkx\" (UniqueName: \"kubernetes.io/projected/b9921584-7832-4cc4-acde-3750a3d8f196-kube-api-access-n7fkx\") pod \"payload-processing-58ddc49fd8-p6cp9\" (UID: \"b9921584-7832-4cc4-acde-3750a3d8f196\") " pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.221345 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.221274 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd2a582-cbbc-4f33-9214-ffbd548bff75-plugins-config-volume\") pod \"payload-pre-processing-cf6c69f79-89fc9\" (UID: \"ecd2a582-cbbc-4f33-9214-ffbd548bff75\") " pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.221940 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.221913 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9921584-7832-4cc4-acde-3750a3d8f196-plugins-config-volume\") pod \"payload-processing-58ddc49fd8-p6cp9\" (UID: \"b9921584-7832-4cc4-acde-3750a3d8f196\") " pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.240348 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.240307 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7fkx\" (UniqueName: \"kubernetes.io/projected/b9921584-7832-4cc4-acde-3750a3d8f196-kube-api-access-n7fkx\") pod \"payload-processing-58ddc49fd8-p6cp9\" (UID: \"b9921584-7832-4cc4-acde-3750a3d8f196\") " pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.322015 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.321936 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd2a582-cbbc-4f33-9214-ffbd548bff75-plugins-config-volume\") pod \"payload-pre-processing-cf6c69f79-89fc9\" (UID: \"ecd2a582-cbbc-4f33-9214-ffbd548bff75\") " pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.322173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.322086 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndv2\" (UniqueName: \"kubernetes.io/projected/ecd2a582-cbbc-4f33-9214-ffbd548bff75-kube-api-access-mndv2\") pod \"payload-pre-processing-cf6c69f79-89fc9\" (UID: \"ecd2a582-cbbc-4f33-9214-ffbd548bff75\") " pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.322583 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.322562 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd2a582-cbbc-4f33-9214-ffbd548bff75-plugins-config-volume\") pod \"payload-pre-processing-cf6c69f79-89fc9\" (UID: \"ecd2a582-cbbc-4f33-9214-ffbd548bff75\") " pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.337538 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.337456 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:35.339423 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.339402 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndv2\" (UniqueName: \"kubernetes.io/projected/ecd2a582-cbbc-4f33-9214-ffbd548bff75-kube-api-access-mndv2\") pod \"payload-pre-processing-cf6c69f79-89fc9\" (UID: \"ecd2a582-cbbc-4f33-9214-ffbd548bff75\") " pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.512614 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.512585 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:35.628551 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.628526 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58ddc49fd8-p6cp9"] Oct 30 04:18:35.630172 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:18:35.630143 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9921584_7832_4cc4_acde_3750a3d8f196.slice/crio-2669ebe973cdd838cd83a83a0bfd5896c401dd0526ef95f2b16a885d823c0b18 WatchSource:0}: Error finding container 2669ebe973cdd838cd83a83a0bfd5896c401dd0526ef95f2b16a885d823c0b18: Status 404 returned error can't find the container with id 2669ebe973cdd838cd83a83a0bfd5896c401dd0526ef95f2b16a885d823c0b18 Oct 30 04:18:35.809234 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:35.809213 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-cf6c69f79-89fc9"] Oct 30 04:18:35.811147 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:18:35.811122 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd2a582_cbbc_4f33_9214_ffbd548bff75.slice/crio-ccc9483b20c0efc76d66e15acb6f629face7b1a66560241122dff530e7cca9e0 WatchSource:0}: Error finding container ccc9483b20c0efc76d66e15acb6f629face7b1a66560241122dff530e7cca9e0: Status 404 returned error can't find the container with id ccc9483b20c0efc76d66e15acb6f629face7b1a66560241122dff530e7cca9e0 Oct 30 04:18:36.585273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.585232 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" event={"ID":"ecd2a582-cbbc-4f33-9214-ffbd548bff75","Type":"ContainerStarted","Data":"6118a90855f4813a6b43b595558aa44e4b3213f2300e1e9077e64ad5f26c45b4"} Oct 30 04:18:36.585765 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.585288 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" event={"ID":"ecd2a582-cbbc-4f33-9214-ffbd548bff75","Type":"ContainerStarted","Data":"ccc9483b20c0efc76d66e15acb6f629face7b1a66560241122dff530e7cca9e0"} Oct 30 04:18:36.585765 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.585318 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:36.586830 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.586804 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" event={"ID":"b9921584-7832-4cc4-acde-3750a3d8f196","Type":"ContainerStarted","Data":"352ccb5f15f396b89f7bda2f078f04b68a290f7df3126f9ea8f6a1de00181371"} Oct 30 04:18:36.586942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.586860 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" event={"ID":"b9921584-7832-4cc4-acde-3750a3d8f196","Type":"ContainerStarted","Data":"2669ebe973cdd838cd83a83a0bfd5896c401dd0526ef95f2b16a885d823c0b18"} Oct 30 04:18:36.586942 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.586884 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:36.622200 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.622160 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" podStartSLOduration=1.622149206 podStartE2EDuration="1.622149206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:36.622091919 +0000 UTC m=+4540.624445115" watchObservedRunningTime="2026-10-30 04:18:36.622149206 +0000 UTC m=+4540.624502402" Oct 30 04:18:36.658656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:36.658608 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" podStartSLOduration=1.658594423 podStartE2EDuration="1.658594423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:36.658416308 +0000 UTC m=+4540.660769505" watchObservedRunningTime="2026-10-30 04:18:36.658594423 +0000 UTC m=+4540.660947621" Oct 30 04:18:38.058984 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.058928 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" podUID="9f652e0b-e10a-47c6-971b-080c5b78b53a" containerName="payload-processing" containerID="cri-o://69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b" gracePeriod=25 Oct 30 04:18:38.300855 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.300832 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 04:18:38.346793 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.346707 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f652e0b-e10a-47c6-971b-080c5b78b53a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f652e0b-e10a-47c6-971b-080c5b78b53a-plugins-config-volume\") pod \"9f652e0b-e10a-47c6-971b-080c5b78b53a\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " Oct 30 04:18:38.346958 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.346918 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f652e0b-e10a-47c6-971b-080c5b78b53a-kube-api-access-skmdc\" (UniqueName: \"kubernetes.io/projected/9f652e0b-e10a-47c6-971b-080c5b78b53a-kube-api-access-skmdc\") pod \"9f652e0b-e10a-47c6-971b-080c5b78b53a\" (UID: \"9f652e0b-e10a-47c6-971b-080c5b78b53a\") " Oct 30 04:18:38.347117 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.347095 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f652e0b-e10a-47c6-971b-080c5b78b53a-plugins-config-volume" pod "9f652e0b-e10a-47c6-971b-080c5b78b53a" (UID: "9f652e0b-e10a-47c6-971b-080c5b78b53a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:38.347231 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.347216 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f652e0b-e10a-47c6-971b-080c5b78b53a-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:18:38.349084 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.349058 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f652e0b-e10a-47c6-971b-080c5b78b53a-kube-api-access-skmdc" pod "9f652e0b-e10a-47c6-971b-080c5b78b53a" (UID: "9f652e0b-e10a-47c6-971b-080c5b78b53a"). InnerVolumeSpecName "kube-api-access-skmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:38.448342 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.448309 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skmdc\" (UniqueName: \"kubernetes.io/projected/9f652e0b-e10a-47c6-971b-080c5b78b53a-kube-api-access-skmdc\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:18:38.596442 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.596405 4442 generic.go:356] "Generic (PLEG): container finished" podID="9f652e0b-e10a-47c6-971b-080c5b78b53a" containerID="69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b" exitCode=0 Oct 30 04:18:38.596607 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.596458 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" Oct 30 04:18:38.596607 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.596492 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" event={"ID":"9f652e0b-e10a-47c6-971b-080c5b78b53a","Type":"ContainerDied","Data":"69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b"} Oct 30 04:18:38.596607 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.596547 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-hgcd9" event={"ID":"9f652e0b-e10a-47c6-971b-080c5b78b53a","Type":"ContainerDied","Data":"cae6d5289a35c456dd0af09d2737bf7192c735b7f09d902ed9903ecc4b0b16cb"} Oct 30 04:18:38.596607 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.596571 4442 scope.go:122] "RemoveContainer" containerID="69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b" Oct 30 04:18:38.605254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.605238 4442 scope.go:122] "RemoveContainer" containerID="69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b" Oct 30 04:18:38.605511 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:18:38.605493 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b\": container with ID starting with 69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b not found: ID does not exist" containerID="69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b" Oct 30 04:18:38.605557 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.605520 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b"} err="failed to get container status \"69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b\": rpc error: code = NotFound desc = could not find container \"69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b\": container with ID starting with 69c1e03cd53ca5567f0a3253217a0abeaca5404b902b4e544d72cafd5ef7462b not found: ID does not exist" Oct 30 04:18:38.642842 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.642816 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hgcd9"] Oct 30 04:18:38.653520 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:38.653499 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-hgcd9"] Oct 30 04:18:40.105827 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:40.105733 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f652e0b-e10a-47c6-971b-080c5b78b53a" path="/var/lib/kubelet/pods/9f652e0b-e10a-47c6-971b-080c5b78b53a/volumes" Oct 30 04:18:43.625881 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:43.625837 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:43.716939 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:43.716904 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:47.594779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:47.594737 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-cf6c69f79-89fc9" Oct 30 04:18:47.595208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:47.594816 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58ddc49fd8-p6cp9" Oct 30 04:18:50.002810 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:50.002772 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:50.186735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:50.186703 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:50.845941 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:50.845907 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:51.113644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.113562 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-v7xhl"] Oct 30 04:18:51.114085 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.113965 4442 cpu_manager.go:414] "containerMap: removing container" podUID="9f652e0b-e10a-47c6-971b-080c5b78b53a" containerName="payload-processing" Oct 30 04:18:51.114085 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.113977 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f652e0b-e10a-47c6-971b-080c5b78b53a" containerName="payload-processing" Oct 30 04:18:51.114085 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.114039 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f652e0b-e10a-47c6-971b-080c5b78b53a" containerName="payload-processing" Oct 30 04:18:51.118032 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.118011 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.134417 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.134395 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Oct 30 04:18:51.136584 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.136558 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-v7xhl"] Oct 30 04:18:51.145882 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.145858 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-ftl9m\"" Oct 30 04:18:51.146017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.145929 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 04:18:51.146017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.145947 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 04:18:51.146017 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.145956 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 04:18:51.146202 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.146024 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 04:18:51.169511 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.169481 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.169649 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.169523 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.169649 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.169615 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-provider-credentials\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.169786 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.169745 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8pp\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-kube-api-access-rh8pp\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.271525 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.271485 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.271826 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.271799 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.271986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.271844 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-provider-credentials\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.271986 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.271909 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8pp\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-kube-api-access-rh8pp\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.272485 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.272454 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.272925 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.272904 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.279692 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.279504 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-provider-credentials\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.292112 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.292072 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8pp\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-kube-api-access-rh8pp\") pod \"payload-processing-external-model-758758b45d-v7xhl\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.431077 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.430997 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:51.732493 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:51.732467 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-v7xhl"] Oct 30 04:18:51.734199 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:18:51.734166 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244170c6_a71e_419f_9b21_02d8e8a2a439.slice/crio-35e5b2a6d2b53d7ffb294aeceea23e0960cc5afaa136a05a35388adc03a5e58b WatchSource:0}: Error finding container 35e5b2a6d2b53d7ffb294aeceea23e0960cc5afaa136a05a35388adc03a5e58b: Status 404 returned error can't find the container with id 35e5b2a6d2b53d7ffb294aeceea23e0960cc5afaa136a05a35388adc03a5e58b Oct 30 04:18:52.373363 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:52.373319 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:18:52.659069 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:52.658986 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" event={"ID":"244170c6-a71e-419f-9b21-02d8e8a2a439","Type":"ContainerStarted","Data":"da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e"} Oct 30 04:18:52.659069 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:52.659047 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" event={"ID":"244170c6-a71e-419f-9b21-02d8e8a2a439","Type":"ContainerStarted","Data":"35e5b2a6d2b53d7ffb294aeceea23e0960cc5afaa136a05a35388adc03a5e58b"} Oct 30 04:18:52.659256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:52.659074 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:18:52.696313 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:52.696270 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" podStartSLOduration=1.696256785 podStartE2EDuration="1.696256785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:52.695326298 +0000 UTC m=+4556.697679495" watchObservedRunningTime="2026-10-30 04:18:52.696256785 +0000 UTC m=+4556.698609982" Oct 30 04:18:53.842778 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:53.841800 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:54.849657 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:54.849620 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:54.964787 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:18:54.960912 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:18:55.185829 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:55.184920 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:18:55.185829 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:55.185001 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:18:55.450995 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:55.450883 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:18:55.451299 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:55.451249 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:18:55.466449 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:55.466421 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:18:55.466702 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:55.466668 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:18:56.643966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:18:56.643935 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:03.669076 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:19:03.669048 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:19:24.859678 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:19:24.859632 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:19:53.940318 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:19:53.940284 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:54.859544 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:19:54.859496 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:19:54.981609 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:19:54.981550 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:56.758357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:19:56.758316 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:24.860123 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:20:24.860038 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:20:40.213483 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:40.213449 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-v7xhl"] Oct 30 04:20:40.277873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:40.277825 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:40.377535 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:40.377494 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:40.568170 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:40.568119 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:41.642697 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:41.642656 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:20:45.214465 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.214400 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" podUID="244170c6-a71e-419f-9b21-02d8e8a2a439" containerName="payload-processing" containerID="cri-o://da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e" gracePeriod=25 Oct 30 04:20:45.471110 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.471059 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:20:45.518256 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.518226 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-plugins-config-volume\") pod \"244170c6-a71e-419f-9b21-02d8e8a2a439\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " Oct 30 04:20:45.518399 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.518330 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-provider-credentials\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-provider-credentials\") pod \"244170c6-a71e-419f-9b21-02d8e8a2a439\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " Oct 30 04:20:45.518399 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.518372 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-kube-api-access-rh8pp\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-kube-api-access-rh8pp\") pod \"244170c6-a71e-419f-9b21-02d8e8a2a439\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " Oct 30 04:20:45.518509 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.518405 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-routing-overlay-volume\") pod \"244170c6-a71e-419f-9b21-02d8e8a2a439\" (UID: \"244170c6-a71e-419f-9b21-02d8e8a2a439\") " Oct 30 04:20:45.518873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.518839 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-plugins-config-volume" pod "244170c6-a71e-419f-9b21-02d8e8a2a439" (UID: "244170c6-a71e-419f-9b21-02d8e8a2a439"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:45.519007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.518952 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-routing-overlay-volume" pod "244170c6-a71e-419f-9b21-02d8e8a2a439" (UID: "244170c6-a71e-419f-9b21-02d8e8a2a439"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:20:45.520728 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.520699 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-kube-api-access-rh8pp" pod "244170c6-a71e-419f-9b21-02d8e8a2a439" (UID: "244170c6-a71e-419f-9b21-02d8e8a2a439"). InnerVolumeSpecName "kube-api-access-rh8pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:45.520728 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.520714 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-provider-credentials" pod "244170c6-a71e-419f-9b21-02d8e8a2a439" (UID: "244170c6-a71e-419f-9b21-02d8e8a2a439"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:45.619257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.619226 4442 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-plugins-config-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.619257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.619253 4442 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-provider-credentials\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.619257 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.619262 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8pp\" (UniqueName: \"kubernetes.io/projected/244170c6-a71e-419f-9b21-02d8e8a2a439-kube-api-access-rh8pp\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:20:45.619493 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:45.619270 4442 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/244170c6-a71e-419f-9b21-02d8e8a2a439-routing-overlay-volume\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:20:46.172450 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.172421 4442 generic.go:356] "Generic (PLEG): container finished" podID="244170c6-a71e-419f-9b21-02d8e8a2a439" containerID="da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e" exitCode=0 Oct 30 04:20:46.172604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.172468 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" Oct 30 04:20:46.172604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.172498 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" event={"ID":"244170c6-a71e-419f-9b21-02d8e8a2a439","Type":"ContainerDied","Data":"da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e"} Oct 30 04:20:46.172604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.172539 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-v7xhl" event={"ID":"244170c6-a71e-419f-9b21-02d8e8a2a439","Type":"ContainerDied","Data":"35e5b2a6d2b53d7ffb294aeceea23e0960cc5afaa136a05a35388adc03a5e58b"} Oct 30 04:20:46.172604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.172563 4442 scope.go:122] "RemoveContainer" containerID="da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e" Oct 30 04:20:46.180990 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.180972 4442 scope.go:122] "RemoveContainer" containerID="da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e" Oct 30 04:20:46.181271 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:20:46.181254 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e\": container with ID starting with da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e not found: ID does not exist" containerID="da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e" Oct 30 04:20:46.181348 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.181281 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e"} err="failed to get container status \"da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e\": rpc error: code = NotFound desc = could not find container \"da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e\": container with ID starting with da397f91cc4b902be4ff74d153cfc132a8bb67b583e2bce9274844db6a34814e not found: ID does not exist" Oct 30 04:20:46.217699 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.217672 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-v7xhl"] Oct 30 04:20:46.228799 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:46.228777 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-v7xhl"] Oct 30 04:20:48.106022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:48.105989 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244170c6-a71e-419f-9b21-02d8e8a2a439" path="/var/lib/kubelet/pods/244170c6-a71e-419f-9b21-02d8e8a2a439/volumes" Oct 30 04:20:48.373955 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:48.373863 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:20:54.057943 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:54.057908 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:54.859421 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:20:54.859387 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:20:55.104473 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:55.104435 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:56.539022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:56.538988 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:56.884796 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:20:56.884697 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:04.082869 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:21:04.082831 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:21:08.877769 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:21:08.877704 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:21:24.860230 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:21:24.860191 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:21:25.768482 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:21:25.768450 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:21:39.974165 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:21:39.974078 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:21:54.163286 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:21:54.163252 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:54.859691 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:21:54.859648 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:21:55.216192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:21:55.216098 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:56.982297 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:21:56.982267 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:03.538020 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:03.537987 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:22:13.747676 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:13.747639 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:22:18.466226 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:18.466190 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:22:24.859583 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:22:24.859541 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:22:35.560639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:35.560602 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:22:40.453779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:40.452848 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:22:40.453779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:40.452994 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:22:40.469787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:40.469197 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:22:40.469787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:40.469272 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:22:40.985780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:40.985012 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:22:45.336166 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:45.336125 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:22:54.288216 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:54.288178 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:54.862641 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:22:54.862595 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:22:55.374202 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:55.374174 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:57.046839 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:57.046807 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:22:57.069765 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:57.069725 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:22:57.071935 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:57.071909 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:22:57.109168 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:57.109143 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:59.971779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:59.971396 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:22:59.971779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:22:59.971463 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:23:02.453199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:02.453163 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:23:10.561170 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:10.561076 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:23:15.951604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:15.951561 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:23:24.860617 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:23:24.860577 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:23:34.255515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:34.255481 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:23:40.184912 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:40.184871 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:23:40.185604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:40.185553 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:23:40.220790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:40.204982 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:23:52.854012 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:52.853974 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:23:54.388362 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:54.388328 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:54.860809 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:23:54.860768 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:23:55.443689 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:55.443650 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:57.222096 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:57.222061 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:58.126744 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:23:58.126713 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:24:03.822039 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:03.821983 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:24:21.831495 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:21.831458 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:24:24.863734 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:24:24.863696 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:24:32.150642 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:32.150611 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:24:37.224537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:37.224502 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:49.541252 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:49.541162 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:24:52.611850 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:52.611816 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:53.131352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:53.131309 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:53.300026 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:53.299984 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:54.481814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:54.481782 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:54.860876 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:24:54.860834 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:24:55.454504 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:55.454469 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:24:55.554268 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:55.554236 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:56.345794 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:56.345734 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:56.444501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:56.444469 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:57.334214 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.334180 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:57.703526 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.703433 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv"] Oct 30 04:24:57.703950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.703929 4442 cpu_manager.go:414] "containerMap: removing container" podUID="244170c6-a71e-419f-9b21-02d8e8a2a439" containerName="payload-processing" Oct 30 04:24:57.703950 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.703947 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244170c6-a71e-419f-9b21-02d8e8a2a439" containerName="payload-processing" Oct 30 04:24:57.704204 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.704187 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="244170c6-a71e-419f-9b21-02d8e8a2a439" containerName="payload-processing" Oct 30 04:24:57.707683 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.707663 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.720851 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.720826 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a9dee1-serving-cert\"" Oct 30 04:24:57.734254 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.734223 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv"] Oct 30 04:24:57.778487 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.778450 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:57.865471 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.865428 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.865658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.865478 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9t5g\" (UniqueName: \"kubernetes.io/projected/b7db7384-9df3-484e-a2fa-9672a20ce5f9-kube-api-access-g9t5g\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.865658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.865548 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.865740 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.865680 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db7384-9df3-484e-a2fa-9672a20ce5f9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.919398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.919341 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:57.967535 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.967452 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.967535 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.967504 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9t5g\" (UniqueName: \"kubernetes.io/projected/b7db7384-9df3-484e-a2fa-9672a20ce5f9-kube-api-access-g9t5g\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.967814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.967549 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.967814 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.967741 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db7384-9df3-484e-a2fa-9672a20ce5f9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.968507 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.968485 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db7384-9df3-484e-a2fa-9672a20ce5f9-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.970133 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.970111 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.970276 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.970248 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:57.993478 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:57.993443 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9t5g\" (UniqueName: \"kubernetes.io/projected/b7db7384-9df3-484e-a2fa-9672a20ce5f9-kube-api-access-g9t5g\") pod \"maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:58.021960 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:58.021928 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:58.095862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:58.095831 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:58.196200 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:58.196171 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:58.336137 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:58.325312 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv"] Oct 30 04:24:58.354786 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:24:58.353705 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7db7384_9df3_484e_a2fa_9672a20ce5f9.slice/crio-6430603c9fa17971445561c824e88704fdfcfbac7548c45eab8c0f938cd6c1e8 WatchSource:0}: Error finding container 6430603c9fa17971445561c824e88704fdfcfbac7548c45eab8c0f938cd6c1e8: Status 404 returned error can't find the container with id 6430603c9fa17971445561c824e88704fdfcfbac7548c45eab8c0f938cd6c1e8 Oct 30 04:24:58.355192 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:58.355165 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:24:58.611959 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:58.611876 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" event={"ID":"b7db7384-9df3-484e-a2fa-9672a20ce5f9","Type":"ContainerStarted","Data":"6430603c9fa17971445561c824e88704fdfcfbac7548c45eab8c0f938cd6c1e8"} Oct 30 04:24:59.554627 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:59.554585 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:59.618067 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:59.618031 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" event={"ID":"b7db7384-9df3-484e-a2fa-9672a20ce5f9","Type":"ContainerStarted","Data":"6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73"} Oct 30 04:24:59.618255 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:59.618138 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:24:59.655358 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:24:59.655308 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" podStartSLOduration=2.267896105 podStartE2EDuration="2.655295016s" totalImagesPullingTime="387.398911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:59.654321023 +0000 UTC m=+4923.656674220" watchObservedRunningTime="2026-10-30 04:24:59.655295016 +0000 UTC m=+4923.657648204" Oct 30 04:25:00.986552 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:00.986514 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:01.969029 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:01.968973 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:25:02.585113 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:02.585079 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:04.084687 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:04.084654 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:05.482924 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:05.482890 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:05.635165 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:05.635135 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:25:07.081891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:07.081843 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:08.575201 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:08.575141 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:10.217518 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:10.217474 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:11.343708 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:11.343667 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:11.726341 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:11.726256 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:12.026137 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:12.026107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:12.129745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:12.129711 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:12.226375 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:12.226337 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:13.288593 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:13.288562 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:13.823675 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:13.823641 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:14.879951 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:14.879916 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:16.472167 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:16.472133 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:25:16.501323 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:16.501294 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:17.880130 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:17.880095 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:17.981455 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:17.981429 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:18.084853 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:18.084818 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:19.589548 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:19.589515 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:21.192679 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:21.192642 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:22.690857 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:22.690817 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:24.292803 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:24.292739 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:24.860027 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:25:24.859994 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:25:25.804691 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:25.804661 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:27.296206 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:27.296163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:28.799013 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:28.798979 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:30.397143 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:30.397108 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:31.963691 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:31.963651 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:25:32.001584 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:32.001545 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:33.597063 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:33.597025 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:35.132960 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:35.132929 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:36.295049 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:36.295019 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:36.392885 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:36.392853 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:36.610306 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:36.610229 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:38.206508 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:38.206473 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:40.568057 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:40.568016 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:41.206637 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:41.206607 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:42.687834 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:42.687803 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:44.180724 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:44.180683 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:45.690126 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:45.690081 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:47.210504 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:47.210470 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:48.693463 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:48.693432 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:50.183774 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:50.183725 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:51.686309 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:51.686256 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:53.187737 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:53.187707 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:54.613025 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:54.612978 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:54.713356 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:54.713317 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:54.859781 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:25:54.859726 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:25:55.666214 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:55.666182 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:56.177914 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:56.174099 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:56.815269 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:56.815234 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:57.301040 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:57.300964 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:57.450745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:57.450711 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:57.587525 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:57.587441 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:58.369936 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:58.369885 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:25:59.082586 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:25:59.082551 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:00.690745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:00.690713 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:01.834829 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:01.834791 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:02.191180 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:02.191061 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:03.536465 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:03.536435 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:03.804163 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:03.804072 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:03.931374 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:03.931333 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:04.050642 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:04.050603 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:05.108462 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.108429 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv"] Oct 30 04:26:05.108907 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.108633 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" podUID="b7db7384-9df3-484e-a2fa-9672a20ce5f9" containerName="maas-api" containerID="cri-o://6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73" gracePeriod=30 Oct 30 04:26:05.152061 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.152012 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:05.595723 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.595697 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:26:05.666588 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.666521 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-metrics-certs\") pod \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " Oct 30 04:26:05.666735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.666593 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7db7384-9df3-484e-a2fa-9672a20ce5f9-kube-api-access-g9t5g\" (UniqueName: \"kubernetes.io/projected/b7db7384-9df3-484e-a2fa-9672a20ce5f9-kube-api-access-g9t5g\") pod \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " Oct 30 04:26:05.666735 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.666637 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-maas-api-tls\") pod \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " Oct 30 04:26:05.666891 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.666780 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7db7384-9df3-484e-a2fa-9672a20ce5f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db7384-9df3-484e-a2fa-9672a20ce5f9-openshift-service-ca\") pod \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\" (UID: \"b7db7384-9df3-484e-a2fa-9672a20ce5f9\") " Oct 30 04:26:05.667139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.667114 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7db7384-9df3-484e-a2fa-9672a20ce5f9-openshift-service-ca" pod "b7db7384-9df3-484e-a2fa-9672a20ce5f9" (UID: "b7db7384-9df3-484e-a2fa-9672a20ce5f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:05.668851 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.668825 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7db7384-9df3-484e-a2fa-9672a20ce5f9-kube-api-access-g9t5g" pod "b7db7384-9df3-484e-a2fa-9672a20ce5f9" (UID: "b7db7384-9df3-484e-a2fa-9672a20ce5f9"). InnerVolumeSpecName "kube-api-access-g9t5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:05.669329 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.669310 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-metrics-certs" pod "b7db7384-9df3-484e-a2fa-9672a20ce5f9" (UID: "b7db7384-9df3-484e-a2fa-9672a20ce5f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:26:05.669406 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.669332 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-maas-api-tls" pod "b7db7384-9df3-484e-a2fa-9672a20ce5f9" (UID: "b7db7384-9df3-484e-a2fa-9672a20ce5f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:26:05.768159 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.768128 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:26:05.768159 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.768155 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9t5g\" (UniqueName: \"kubernetes.io/projected/b7db7384-9df3-484e-a2fa-9672a20ce5f9-kube-api-access-g9t5g\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:26:05.768159 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.768165 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7db7384-9df3-484e-a2fa-9672a20ce5f9-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:26:05.768394 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.768176 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7db7384-9df3-484e-a2fa-9672a20ce5f9-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:26:05.945934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.945845 4442 generic.go:356] "Generic (PLEG): container finished" podID="b7db7384-9df3-484e-a2fa-9672a20ce5f9" containerID="6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73" exitCode=0 Oct 30 04:26:05.945934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.945898 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" Oct 30 04:26:05.946139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.945931 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" event={"ID":"b7db7384-9df3-484e-a2fa-9672a20ce5f9","Type":"ContainerDied","Data":"6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73"} Oct 30 04:26:05.946139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.945992 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv" event={"ID":"b7db7384-9df3-484e-a2fa-9672a20ce5f9","Type":"ContainerDied","Data":"6430603c9fa17971445561c824e88704fdfcfbac7548c45eab8c0f938cd6c1e8"} Oct 30 04:26:05.946139 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.946015 4442 scope.go:122] "RemoveContainer" containerID="6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73" Oct 30 04:26:05.955375 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.955356 4442 scope.go:122] "RemoveContainer" containerID="6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73" Oct 30 04:26:05.955630 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:26:05.955608 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73\": container with ID starting with 6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73 not found: ID does not exist" containerID="6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73" Oct 30 04:26:05.955740 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.955638 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73"} err="failed to get container status \"6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73\": rpc error: code = NotFound desc = could not find container \"6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73\": container with ID starting with 6088cdee612c707f76b47cbc4c6c534e5812bd254239590e503cee32aba17c73 not found: ID does not exist" Oct 30 04:26:05.991706 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:05.991683 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv"] Oct 30 04:26:06.002926 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:06.002905 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a9dee1-5dc95444c8-rn9dv"] Oct 30 04:26:06.105242 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:06.105216 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7db7384-9df3-484e-a2fa-9672a20ce5f9" path="/var/lib/kubelet/pods/b7db7384-9df3-484e-a2fa-9672a20ce5f9/volumes" Oct 30 04:26:07.668817 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:07.668742 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:26:09.515928 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:09.515838 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:09.616135 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:09.616107 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:10.221241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:10.221213 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:10.604948 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:10.604900 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:24.226497 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:24.226461 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:26:24.954779 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:26:24.953511 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:26:28.967748 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:28.967715 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:38.261496 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:38.261461 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:39.195841 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:39.195802 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:44.818479 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:44.818444 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:50.399183 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:50.399148 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:54.723270 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:54.723236 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:54.859944 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:26:54.859907 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:26:55.779244 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:55.779198 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:57.571862 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:26:57.571830 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:24.860626 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:27:24.860584 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:27:52.758559 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:52.758481 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:27:54.938398 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:54.938303 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:54.954485 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:27:54.954416 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:27:55.199976 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:55.199846 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:27:55.200302 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:55.200266 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:27:55.446886 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:55.446845 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:27:55.447179 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:55.447150 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:27:55.475946 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:55.475857 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:27:55.476203 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:55.476173 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:27:55.900460 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:55.900428 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:57.072435 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:57.072407 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:27:57.097300 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:57.097279 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:27:57.687419 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:27:57.687387 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:08.155400 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:08.155366 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:28:13.314779 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:13.303928 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:28:17.681078 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:17.681037 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:28:24.859425 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:28:24.859387 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:28:26.657576 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.657533 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7"] Oct 30 04:28:26.658288 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.658125 4442 cpu_manager.go:414] "containerMap: removing container" podUID="b7db7384-9df3-484e-a2fa-9672a20ce5f9" containerName="maas-api" Oct 30 04:28:26.658288 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.658146 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7db7384-9df3-484e-a2fa-9672a20ce5f9" containerName="maas-api" Oct 30 04:28:26.658288 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.658244 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7db7384-9df3-484e-a2fa-9672a20ce5f9" containerName="maas-api" Oct 30 04:28:26.661424 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.661403 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.690019 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.689989 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7"] Oct 30 04:28:26.774237 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.774208 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4289e4da-a1da-4115-abd2-558d6c298f85-metrics-certs\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.774428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.774270 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4289e4da-a1da-4115-abd2-558d6c298f85-openshift-service-ca\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.774428 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.774413 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4289e4da-a1da-4115-abd2-558d6c298f85-maas-api-tls\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.774528 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.774446 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4gw\" (UniqueName: \"kubernetes.io/projected/4289e4da-a1da-4115-abd2-558d6c298f85-kube-api-access-6d4gw\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.875418 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.875375 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4289e4da-a1da-4115-abd2-558d6c298f85-maas-api-tls\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.875612 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.875430 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4gw\" (UniqueName: \"kubernetes.io/projected/4289e4da-a1da-4115-abd2-558d6c298f85-kube-api-access-6d4gw\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.875612 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.875490 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4289e4da-a1da-4115-abd2-558d6c298f85-metrics-certs\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.875612 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.875551 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4289e4da-a1da-4115-abd2-558d6c298f85-openshift-service-ca\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.876315 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.876289 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4289e4da-a1da-4115-abd2-558d6c298f85-openshift-service-ca\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.878173 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.878141 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4289e4da-a1da-4115-abd2-558d6c298f85-metrics-certs\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.878279 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.878204 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4289e4da-a1da-4115-abd2-558d6c298f85-maas-api-tls\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.893444 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.893419 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4gw\" (UniqueName: \"kubernetes.io/projected/4289e4da-a1da-4115-abd2-558d6c298f85-kube-api-access-6d4gw\") pod \"maas-api-bcb58fd6d-w9rg7\" (UID: \"4289e4da-a1da-4115-abd2-558d6c298f85\") " pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:26.976658 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:26.976555 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:27.274966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:27.274941 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7"] Oct 30 04:28:27.276975 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:28:27.276946 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4289e4da_a1da_4115_abd2_558d6c298f85.slice/crio-f9791e6d69f5831bc1c7090a23e55e7a5b60bc992bac665cfa95dd6c3e17657c WatchSource:0}: Error finding container f9791e6d69f5831bc1c7090a23e55e7a5b60bc992bac665cfa95dd6c3e17657c: Status 404 returned error can't find the container with id f9791e6d69f5831bc1c7090a23e55e7a5b60bc992bac665cfa95dd6c3e17657c Oct 30 04:28:27.631618 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:27.631539 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" event={"ID":"4289e4da-a1da-4115-abd2-558d6c298f85","Type":"ContainerStarted","Data":"f9791e6d69f5831bc1c7090a23e55e7a5b60bc992bac665cfa95dd6c3e17657c"} Oct 30 04:28:28.637377 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:28.637337 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" event={"ID":"4289e4da-a1da-4115-abd2-558d6c298f85","Type":"ContainerStarted","Data":"075f07263606ab66634196b31aee79fb6c219c97b9c95cb55e4f0218d208e611"} Oct 30 04:28:28.637812 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:28.637416 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:28.676087 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:28.676034 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" podStartSLOduration=2.297184514 podStartE2EDuration="2.676022044s" totalImagesPullingTime="378.83753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:28.674690838 +0000 UTC m=+5132.677044073" watchObservedRunningTime="2026-10-30 04:28:28.676022044 +0000 UTC m=+5132.678375253" Oct 30 04:28:34.647686 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:34.647654 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bcb58fd6d-w9rg7" Oct 30 04:28:34.717873 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:34.717828 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js"] Oct 30 04:28:34.718095 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:34.718068 4442 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" podUID="987ddc6c-c347-4ba8-b6ee-6e80ba285321" containerName="maas-api" containerID="cri-o://683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282" gracePeriod=30 Oct 30 04:28:35.959483 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:35.959460 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 04:28:36.057836 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.057747 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-maas-api-tls\") pod \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " Oct 30 04:28:36.057996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.057858 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987ddc6c-c347-4ba8-b6ee-6e80ba285321-kube-api-access-txnvg\" (UniqueName: \"kubernetes.io/projected/987ddc6c-c347-4ba8-b6ee-6e80ba285321-kube-api-access-txnvg\") pod \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " Oct 30 04:28:36.057996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.057905 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-metrics-certs\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-metrics-certs\") pod \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " Oct 30 04:28:36.057996 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.057955 4442 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987ddc6c-c347-4ba8-b6ee-6e80ba285321-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ddc6c-c347-4ba8-b6ee-6e80ba285321-openshift-service-ca\") pod \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\" (UID: \"987ddc6c-c347-4ba8-b6ee-6e80ba285321\") " Oct 30 04:28:36.058366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.058331 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987ddc6c-c347-4ba8-b6ee-6e80ba285321-openshift-service-ca" pod "987ddc6c-c347-4ba8-b6ee-6e80ba285321" (UID: "987ddc6c-c347-4ba8-b6ee-6e80ba285321"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:28:36.059985 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.059956 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-maas-api-tls" pod "987ddc6c-c347-4ba8-b6ee-6e80ba285321" (UID: "987ddc6c-c347-4ba8-b6ee-6e80ba285321"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:28:36.060087 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.060046 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987ddc6c-c347-4ba8-b6ee-6e80ba285321-kube-api-access-txnvg" pod "987ddc6c-c347-4ba8-b6ee-6e80ba285321" (UID: "987ddc6c-c347-4ba8-b6ee-6e80ba285321"). InnerVolumeSpecName "kube-api-access-txnvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:28:36.060137 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.060119 4442 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-metrics-certs" pod "987ddc6c-c347-4ba8-b6ee-6e80ba285321" (UID: "987ddc6c-c347-4ba8-b6ee-6e80ba285321"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:28:36.159787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.159745 4442 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/987ddc6c-c347-4ba8-b6ee-6e80ba285321-openshift-service-ca\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:28:36.159932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.159881 4442 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-maas-api-tls\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:28:36.159932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.159903 4442 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txnvg\" (UniqueName: \"kubernetes.io/projected/987ddc6c-c347-4ba8-b6ee-6e80ba285321-kube-api-access-txnvg\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:28:36.159932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.159924 4442 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/987ddc6c-c347-4ba8-b6ee-6e80ba285321-metrics-certs\") on node \"ip-10-0-134-92.ec2.internal\" DevicePath \"\"" Oct 30 04:28:36.673604 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.673569 4442 generic.go:356] "Generic (PLEG): container finished" podID="987ddc6c-c347-4ba8-b6ee-6e80ba285321" containerID="683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282" exitCode=0 Oct 30 04:28:36.673809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.673619 4442 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" Oct 30 04:28:36.673809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.673653 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" event={"ID":"987ddc6c-c347-4ba8-b6ee-6e80ba285321","Type":"ContainerDied","Data":"683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282"} Oct 30 04:28:36.673809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.673710 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js" event={"ID":"987ddc6c-c347-4ba8-b6ee-6e80ba285321","Type":"ContainerDied","Data":"6add5f56c894142eaa51220ec3a2c03959ac240a14afb6c7f614dda22842a4aa"} Oct 30 04:28:36.673809 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.673732 4442 scope.go:122] "RemoveContainer" containerID="683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282" Oct 30 04:28:36.682964 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.682941 4442 scope.go:122] "RemoveContainer" containerID="683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282" Oct 30 04:28:36.683244 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:28:36.683226 4442 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282\": container with ID starting with 683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282 not found: ID does not exist" containerID="683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282" Oct 30 04:28:36.683293 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.683253 4442 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282"} err="failed to get container status \"683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282\": rpc error: code = NotFound desc = could not find container \"683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282\": container with ID starting with 683309b3597ea27bdee11ec183dbd22babc996aed12961f71b13964648768282 not found: ID does not exist" Oct 30 04:28:36.718910 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.718885 4442 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js"] Oct 30 04:28:36.729654 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:36.729631 4442 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65b844bfb4-vk8js"] Oct 30 04:28:37.943305 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:37.943272 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:28:38.110548 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:38.110502 4442 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987ddc6c-c347-4ba8-b6ee-6e80ba285321" path="/var/lib/kubelet/pods/987ddc6c-c347-4ba8-b6ee-6e80ba285321/volumes" Oct 30 04:28:40.181015 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:40.180977 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:28:40.181648 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:40.181615 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:28:40.459049 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:40.458943 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:28:40.459358 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:40.459325 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:28:40.476872 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:40.476840 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:28:40.477081 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:40.477055 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:28:54.860714 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:28:54.860673 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:28:55.070209 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:55.070172 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:56.003662 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:56.003630 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:57.796677 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:28:57.796633 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:06.934899 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:06.934863 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:29:12.125316 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:12.125236 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:29:14.976780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:14.975884 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:29:14.976780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:14.976001 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:29:24.859741 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:29:24.859702 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:29:30.786208 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:30.786169 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:29:35.459762 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:35.459714 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:29:41.191787 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:41.190672 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:29:54.858812 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:29:54.858777 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:29:55.176273 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:55.176165 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:56.128007 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:56.127954 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:57.915674 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:29:57.915636 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:01.443268 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:01.443231 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:30:11.831615 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:11.831571 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:30:17.167992 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:17.167951 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:30:22.425199 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:22.425162 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:30:24.997036 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:30:24.996980 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:30:25.178949 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:25.178892 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-bp56d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:30:25.179295 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:25.179247 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-bp56d" podUID="8894e5e4-3e55-4cae-9400-27794bd21f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:30:25.457780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:25.451857 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:30:25.457780 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:25.451927 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-sq5vm" podUID="2c23bba8-edb0-4f3d-9d05-aca46b96e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:30:25.468932 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:25.468899 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:30:25.469203 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:25.469163 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:30:40.746346 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:40.746210 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:30:51.062901 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:51.062863 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:30:54.959786 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:30:54.957478 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:30:55.378501 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:55.378462 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:56.362478 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:56.362371 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:56.463771 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:56.462963 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:30:58.035526 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:30:58.035491 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:08.130426 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:08.130394 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:31:10.466909 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:10.466864 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:31:10.467572 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:10.467534 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-s6z4q" podUID="2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:31:16.055487 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:16.055441 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:18.372769 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:18.372730 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:24.860833 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:31:24.860795 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:31:24.941044 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:24.941006 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:31:54.859541 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:31:54.859492 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:31:55.504696 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:55.504652 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:56.606276 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:56.606241 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:58.184773 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:58.182066 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:58.594086 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:31:58.594028 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:24.859809 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:32:24.859774 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:32:46.912959 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:46.912924 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:54.863374 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:32:54.863334 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:32:55.591443 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:55.591402 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:56.694745 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:56.694713 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:57.098645 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:57.098611 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:32:57.124262 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:57.124238 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qtbgx_585acb34-e959-4d7b-9e5a-7e1e4fbfccea/ovn-acl-logging/0.log" Oct 30 04:32:58.313022 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:58.312975 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:59.976905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:59.973842 4442 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:32:59.976905 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:32:59.973932 4442 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77bfb58f8c-vtlvz" podUID="d2d45fb8-f876-4f98-9f0f-72bb69aaaa68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:33:17.849547 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:17.849508 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ndw77"] Oct 30 04:33:24.860290 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:33:24.860254 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:33:39.233026 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:39.232938 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:33:40.237977 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:40.237946 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-rwz7m_d83c13ee-a5c0-4a5f-8a33-57aeb805a233/manager/0.log" Oct 30 04:33:42.135108 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:42.135077 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bcb58fd6d-w9rg7_4289e4da-a1da-4115-abd2-558d6c298f85/maas-api/0.log" Oct 30 04:33:49.108600 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:49.108572 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ndw77_874a2d48-0e3f-45c1-bce3-bfccb9d006b8/limitador/0.log" Oct 30 04:33:50.481012 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:50.480984 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tcgcs_f3405637-4232-4d64-b5cd-03b9508a7e41/discovery/0.log" Oct 30 04:33:50.687386 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:50.687359 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-bjqd8_b5b60c17-548f-41cc-9313-5836a6171c81/kube-auth-proxy/0.log" Oct 30 04:33:51.292655 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:51.292623 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-bp56d_8894e5e4-3e55-4cae-9400-27794bd21f0a/istio-proxy/0.log" Oct 30 04:33:52.358991 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:52.358957 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-s6z4q_2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908/istio-proxy/0.log" Oct 30 04:33:52.551621 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:52.551589 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-sq5vm_2c23bba8-edb0-4f3d-9d05-aca46b96e13c/istio-proxy/0.log" Oct 30 04:33:52.749644 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:52.749623 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-vtlvz_d2d45fb8-f876-4f98-9f0f-72bb69aaaa68/istio-proxy/0.log" Oct 30 04:33:53.368235 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:53.368208 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-cf6c69f79-89fc9_ecd2a582-cbbc-4f33-9214-ffbd548bff75/payload-pre-processing/0.log" Oct 30 04:33:53.540101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:53.540069 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58ddc49fd8-p6cp9_b9921584-7832-4cc4-acde-3750a3d8f196/payload-processing/0.log" Oct 30 04:33:54.859475 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:33:54.859441 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:33:54.913687 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:54.913663 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-4wc6k_cdbebe0f-c13d-4fca-ba8a-b3efd2a6815d/main/0.log" Oct 30 04:33:55.491510 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:55.491479 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-k7xc4_6c43243c-5d9f-46c9-89c6-816d34027ccc/main/0.log" Oct 30 04:33:55.691191 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:55.691154 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:55.883304 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:55.883267 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-bcrdv_1f31cbac-b475-488d-9098-f7e3e4a24071/main/0.log" Oct 30 04:33:56.294684 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:56.294656 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-n2v2v_b83c6124-3537-4816-b12a-ae756634bcc6/main/0.log" Oct 30 04:33:56.798934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:56.798904 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:58.294293 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:58.294255 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:58.394906 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:33:58.394874 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:07.365612 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.365565 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:07.383066 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.383040 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lwlgf/must-gather-hzql6"] Oct 30 04:34:07.383466 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.383444 4442 cpu_manager.go:414] "containerMap: removing container" podUID="987ddc6c-c347-4ba8-b6ee-6e80ba285321" containerName="maas-api" Oct 30 04:34:07.383630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.383604 4442 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987ddc6c-c347-4ba8-b6ee-6e80ba285321" containerName="maas-api" Oct 30 04:34:07.383777 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.383742 4442 memory_manager.go:365] "RemoveStaleState removing state" podUID="987ddc6c-c347-4ba8-b6ee-6e80ba285321" containerName="maas-api" Oct 30 04:34:07.387004 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.386987 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.405797 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.405771 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lwlgf\"/\"default-dockercfg-ffh54\"" Oct 30 04:34:07.416496 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.416475 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwlgf\"/\"openshift-service-ca.crt\"" Oct 30 04:34:07.416621 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.416499 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lwlgf\"/\"kube-root-ca.crt\"" Oct 30 04:34:07.419391 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.419367 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwlgf/must-gather-hzql6"] Oct 30 04:34:07.440189 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.440163 4442 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:07.504541 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.504510 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ba3e281-238e-467b-bd29-44337a407019-must-gather-output\") pod \"must-gather-hzql6\" (UID: \"3ba3e281-238e-467b-bd29-44337a407019\") " pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.504726 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.504704 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756zm\" (UniqueName: \"kubernetes.io/projected/3ba3e281-238e-467b-bd29-44337a407019-kube-api-access-756zm\") pod \"must-gather-hzql6\" (UID: \"3ba3e281-238e-467b-bd29-44337a407019\") " pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.605324 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.605293 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756zm\" (UniqueName: \"kubernetes.io/projected/3ba3e281-238e-467b-bd29-44337a407019-kube-api-access-756zm\") pod \"must-gather-hzql6\" (UID: \"3ba3e281-238e-467b-bd29-44337a407019\") " pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.605511 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.605342 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ba3e281-238e-467b-bd29-44337a407019-must-gather-output\") pod \"must-gather-hzql6\" (UID: \"3ba3e281-238e-467b-bd29-44337a407019\") " pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.605701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.605682 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3ba3e281-238e-467b-bd29-44337a407019-must-gather-output\") pod \"must-gather-hzql6\" (UID: \"3ba3e281-238e-467b-bd29-44337a407019\") " pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.626966 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.626911 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756zm\" (UniqueName: \"kubernetes.io/projected/3ba3e281-238e-467b-bd29-44337a407019-kube-api-access-756zm\") pod \"must-gather-hzql6\" (UID: \"3ba3e281-238e-467b-bd29-44337a407019\") " pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.701537 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.701511 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lwlgf/must-gather-hzql6" Oct 30 04:34:07.998311 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:07.998190 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lwlgf/must-gather-hzql6"] Oct 30 04:34:08.000949 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:34:08.000923 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ba3e281_238e_467b_bd29_44337a407019.slice/crio-d221d8028d63f86dc507f2cfbe484b6c5a8c3c657e5bc3e6fff1a9350bc7acb4 WatchSource:0}: Error finding container d221d8028d63f86dc507f2cfbe484b6c5a8c3c657e5bc3e6fff1a9350bc7acb4: Status 404 returned error can't find the container with id d221d8028d63f86dc507f2cfbe484b6c5a8c3c657e5bc3e6fff1a9350bc7acb4 Oct 30 04:34:08.003262 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:08.003245 4442 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:34:08.704215 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:08.704179 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwlgf/must-gather-hzql6" event={"ID":"3ba3e281-238e-467b-bd29-44337a407019","Type":"ContainerStarted","Data":"d221d8028d63f86dc507f2cfbe484b6c5a8c3c657e5bc3e6fff1a9350bc7acb4"} Oct 30 04:34:12.931549 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:12.931511 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m266j_90b350e9-6998-45b6-882a-17019e89bdf6/global-pull-secret-syncer/0.log" Oct 30 04:34:13.734431 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:13.734399 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwlgf/must-gather-hzql6" event={"ID":"3ba3e281-238e-467b-bd29-44337a407019","Type":"ContainerStarted","Data":"38d7951c3249962f7b73462385e58bb527b21338def7624f49029079953b12f6"} Oct 30 04:34:13.734616 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:13.734438 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lwlgf/must-gather-hzql6" event={"ID":"3ba3e281-238e-467b-bd29-44337a407019","Type":"ContainerStarted","Data":"27b556abf14f7e8db54a83622126bc485de10471bc90b248f305f18d988bbbf1"} Oct 30 04:34:13.770515 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:13.770458 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lwlgf/must-gather-hzql6" podStartSLOduration=1.668922898 podStartE2EDuration="6.770444337s" totalImagesPullingTime="5.101521439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:34:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:13.768952949 +0000 UTC m=+5477.771306153" watchObservedRunningTime="2026-10-30 04:34:13.770444337 +0000 UTC m=+5477.772797534" Oct 30 04:34:13.934831 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:13.934797 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j2q75_3fcbb594-f806-4ac7-ab99-ea99ca6c827a/konnectivity-agent/0.log" Oct 30 04:34:14.378272 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:14.378242 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-92.ec2.internal_7c18c0405adfb2995cb6b447bf5eb736/haproxy/0.log" Oct 30 04:34:14.944630 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:14.944567 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:34:15.797447 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:15.797392 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-rwz7m_d83c13ee-a5c0-4a5f-8a33-57aeb805a233/manager/0.log" Oct 30 04:34:18.974889 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:18.974853 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:34:19.903934 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:19.903905 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-rwz7m_d83c13ee-a5c0-4a5f-8a33-57aeb805a233/manager/0.log" Oct 30 04:34:23.471914 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:23.471877 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:34:23.902332 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:23.902300 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ndw77_874a2d48-0e3f-45c1-bce3-bfccb9d006b8/limitador/0.log" Oct 30 04:34:24.412024 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.411990 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-rwz7m_d83c13ee-a5c0-4a5f-8a33-57aeb805a233/manager/0.log" Oct 30 04:34:24.545960 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.545908 4442 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc"] Oct 30 04:34:24.554101 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.554073 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.567894 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.567864 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc"] Oct 30 04:34:24.568322 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.568292 4442 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s9nqc\"/\"default-dockercfg-kgslt\"" Oct 30 04:34:24.575038 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.575016 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nqc\"/\"openshift-service-ca.crt\"" Oct 30 04:34:24.579106 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.579043 4442 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s9nqc\"/\"kube-root-ca.crt\"" Oct 30 04:34:24.686352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.686237 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47w4n\" (UniqueName: \"kubernetes.io/projected/a95bdf76-0a0f-46b1-b228-7624fbcbf055-kube-api-access-47w4n\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.686352 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.686282 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-sys\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.686608 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.686401 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-podres\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.686608 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.686479 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-proc\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.686608 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.686498 4442 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-lib-modules\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788010 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.787967 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-sys\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788056 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-podres\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788110 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-proc\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788116 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-sys\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788130 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-lib-modules\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788208 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-proc\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788206 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-lib-modules\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788273 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a95bdf76-0a0f-46b1-b228-7624fbcbf055-podres\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.788308 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.788329 4442 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47w4n\" (UniqueName: \"kubernetes.io/projected/a95bdf76-0a0f-46b1-b228-7624fbcbf055-kube-api-access-47w4n\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.810590 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.810541 4442 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47w4n\" (UniqueName: \"kubernetes.io/projected/a95bdf76-0a0f-46b1-b228-7624fbcbf055-kube-api-access-47w4n\") pod \"perf-node-gather-daemonset-n8fmc\" (UID: \"a95bdf76-0a0f-46b1-b228-7624fbcbf055\") " pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:24.864425 ip-10-0-134-92 kubenswrapper[4442]: E1030 04:34:24.864390 4442 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-cp2d8" containerName="extractor" Oct 30 04:34:24.873677 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:24.873652 4442 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:25.181328 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:25.181302 4442 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc"] Oct 30 04:34:25.184441 ip-10-0-134-92 kubenswrapper[4442]: W1030 04:34:25.184408 4442 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda95bdf76_0a0f_46b1_b228_7624fbcbf055.slice/crio-5231bf4c6b86d01d64f67a60b2a01fc1170cb90bfcc5f775d496a28acf257f62 WatchSource:0}: Error finding container 5231bf4c6b86d01d64f67a60b2a01fc1170cb90bfcc5f775d496a28acf257f62: Status 404 returned error can't find the container with id 5231bf4c6b86d01d64f67a60b2a01fc1170cb90bfcc5f775d496a28acf257f62 Oct 30 04:34:25.299680 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:25.299593 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:34:25.802399 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:25.802271 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" event={"ID":"a95bdf76-0a0f-46b1-b228-7624fbcbf055","Type":"ContainerStarted","Data":"b36f4fddd8704ef108330a86d7796b55eeaa5dc9d156c78b96813dc8ac8f6ab3"} Oct 30 04:34:25.802399 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:25.802346 4442 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" event={"ID":"a95bdf76-0a0f-46b1-b228-7624fbcbf055","Type":"ContainerStarted","Data":"5231bf4c6b86d01d64f67a60b2a01fc1170cb90bfcc5f775d496a28acf257f62"} Oct 30 04:34:25.802968 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:25.802483 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:25.840706 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:25.840629 4442 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" podStartSLOduration=1.840612734 podStartE2EDuration="1.840612734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:25.838776389 +0000 UTC m=+5489.841129585" watchObservedRunningTime="2026-10-30 04:34:25.840612734 +0000 UTC m=+5489.842965931" Oct 30 04:34:26.541835 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:26.541800 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-rwz7m_d83c13ee-a5c0-4a5f-8a33-57aeb805a233/manager/0.log" Oct 30 04:34:27.769366 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:27.767691 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-db4fc75f6-lzwkl_98c6c174-f353-40f4-9e1b-f4719ea7b024/metrics-server/0.log" Oct 30 04:34:27.892565 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:27.892513 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-v8mhc_3ab9f131-32d6-49d4-8c83-60938c842a3f/monitoring-plugin/0.log" Oct 30 04:34:28.764095 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:28.764058 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8zv7_cd5d7d80-b047-4446-ad12-945bca88164b/node-exporter/0.log" Oct 30 04:34:28.879318 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:28.879287 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8zv7_cd5d7d80-b047-4446-ad12-945bca88164b/kube-rbac-proxy/0.log" Oct 30 04:34:28.995001 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:28.994942 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8zv7_cd5d7d80-b047-4446-ad12-945bca88164b/init-textfile/0.log" Oct 30 04:34:29.481600 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:29.481556 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:34:30.993081 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:30.993043 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tcgcs_f3405637-4232-4d64-b5cd-03b9508a7e41/discovery/0.log" Oct 30 04:34:31.025357 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.025333 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-bjqd8_b5b60c17-548f-41cc-9313-5836a6171c81/kube-auth-proxy/0.log" Oct 30 04:34:31.339856 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.339826 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-rwz7m_d83c13ee-a5c0-4a5f-8a33-57aeb805a233/manager/0.log" Oct 30 04:34:31.355178 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.355145 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-bp56d_8894e5e4-3e55-4cae-9400-27794bd21f0a/istio-proxy/0.log" Oct 30 04:34:31.669383 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.669298 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-s6z4q_2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908/istio-proxy/0.log" Oct 30 04:34:31.686511 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.686481 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bjg6c_f33aee02-cbf6-48d8-b56d-3b8895651792/prometheus-operator/0.log" Oct 30 04:34:31.734385 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.734349 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-sq5vm_2c23bba8-edb0-4f3d-9d05-aca46b96e13c/istio-proxy/0.log" Oct 30 04:34:31.799533 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.799503 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-vtlvz_d2d45fb8-f876-4f98-9f0f-72bb69aaaa68/istio-proxy/0.log" Oct 30 04:34:31.805330 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.805306 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-bjg6c_f33aee02-cbf6-48d8-b56d-3b8895651792/kube-rbac-proxy/0.log" Oct 30 04:34:31.819049 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.819027 4442 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s9nqc/perf-node-gather-daemonset-n8fmc" Oct 30 04:34:31.987992 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:31.987925 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-cf6c69f79-89fc9_ecd2a582-cbbc-4f33-9214-ffbd548bff75/payload-pre-processing/0.log" Oct 30 04:34:32.050129 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:32.050098 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58ddc49fd8-p6cp9_b9921584-7832-4cc4-acde-3750a3d8f196/payload-processing/0.log" Oct 30 04:34:34.576843 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:34.576799 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b5fdc8966-ztpcp_09e1682b-def2-4c93-8bd9-0f90131c27d9/manager/0.log" Oct 30 04:34:36.116625 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:36.116584 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-679796c8cb-rwz7m_d83c13ee-a5c0-4a5f-8a33-57aeb805a233/manager/0.log" Oct 30 04:34:37.943639 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:37.943609 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ndw77_874a2d48-0e3f-45c1-bce3-bfccb9d006b8/limitador/0.log" Oct 30 04:34:41.343243 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:41.343212 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9625v_c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58/dns/0.log" Oct 30 04:34:41.455312 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:41.455272 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-9625v_c63a8507-8ba7-45ac-b6a0-e0e00c7b1a58/kube-rbac-proxy/0.log" Oct 30 04:34:42.354541 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:42.354511 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jj8ks_436fce07-fd6a-4b3e-8b1a-9d2f9de6e6b7/dns-node-resolver/0.log" Oct 30 04:34:42.642725 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:42.642628 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tcgcs_f3405637-4232-4d64-b5cd-03b9508a7e41/discovery/0.log" Oct 30 04:34:42.675640 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:42.675611 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-bjqd8_b5b60c17-548f-41cc-9313-5836a6171c81/kube-auth-proxy/0.log" Oct 30 04:34:42.872109 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:42.872073 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-bp56d_8894e5e4-3e55-4cae-9400-27794bd21f0a/istio-proxy/0.log" Oct 30 04:34:43.173812 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.173747 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-s6z4q_2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908/istio-proxy/0.log" Oct 30 04:34:43.235790 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.235727 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-sq5vm_2c23bba8-edb0-4f3d-9d05-aca46b96e13c/istio-proxy/0.log" Oct 30 04:34:43.302701 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.302671 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-vtlvz_d2d45fb8-f876-4f98-9f0f-72bb69aaaa68/istio-proxy/0.log" Oct 30 04:34:43.490656 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.490572 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-cf6c69f79-89fc9_ecd2a582-cbbc-4f33-9214-ffbd548bff75/payload-pre-processing/0.log" Oct 30 04:34:43.512995 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.512970 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8684848d94-c9w9v_a01652d2-9f9b-4ae6-9988-b27ee8d5952b/registry/0.log" Oct 30 04:34:43.553158 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.553129 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58ddc49fd8-p6cp9_b9921584-7832-4cc4-acde-3750a3d8f196/payload-processing/0.log" Oct 30 04:34:43.741643 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.740524 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-shbmk_50f7ec66-f78a-405d-95c0-47a26e24a69d/node-ca/0.log" Oct 30 04:34:43.978531 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:43.978504 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tcgcs_f3405637-4232-4d64-b5cd-03b9508a7e41/discovery/0.log" Oct 30 04:34:44.010087 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:44.010064 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78ccd95bcf-bjqd8_b5b60c17-548f-41cc-9313-5836a6171c81/kube-auth-proxy/0.log" Oct 30 04:34:44.189127 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:44.189093 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-bp56d_8894e5e4-3e55-4cae-9400-27794bd21f0a/istio-proxy/0.log" Oct 30 04:34:44.491011 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:44.490929 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-s6z4q_2dfb7b27-4cc9-4c1b-8eeb-ce44961a4908/istio-proxy/0.log" Oct 30 04:34:44.553247 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:44.553215 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-sq5vm_2c23bba8-edb0-4f3d-9d05-aca46b96e13c/istio-proxy/0.log" Oct 30 04:34:44.620241 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:44.620217 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77bfb58f8c-vtlvz_d2d45fb8-f876-4f98-9f0f-72bb69aaaa68/istio-proxy/0.log" Oct 30 04:34:44.800283 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:44.800259 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-cf6c69f79-89fc9_ecd2a582-cbbc-4f33-9214-ffbd548bff75/payload-pre-processing/0.log" Oct 30 04:34:44.857998 ip-10-0-134-92 kubenswrapper[4442]: I1030 04:34:44.857964 4442 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58ddc49fd8-p6cp9_b9921584-7832-4cc4-acde-3750a3d8f196/payload-processing/0.log"